Max level shown:
1 [ Oct 1 02:30:10 Enabled. ]
2 [ Oct 1 02:30:10 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Oct 1 02:30:10 Method "start" exited with status 0. ]
4 [ Oct 1 02:30:10 Rereading configuration. ]
5 [ Oct 1 02:30:10 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-10-01T02:30:10.474ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-10-01T02:30:10.474ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-10-01T02:30:10.528ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-10-01T02:30:10.528ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-10-01T02:30:10.640ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-10-01T02:30:10.649ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-10-01T02:30:10.657ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-10-01T02:30:11.261ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-10-01T02:30:11.399ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-10-01T02:30:11.399ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-10-01T02:30:11.399ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-10-01T02:30:11.399ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-10-01T02:30:11.399ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-10-01T02:30:11.399ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-10-01T02:30:11.402ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-10-01T02:30:11.402ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-10-01T02:30:11.402ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-10-01T02:30:11.402ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-10-01T02:30:11.402ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-10-01T02:30:11.403ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-10-01T02:30:11.403ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-10-01T02:30:11.403ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-10-01T02:30:11.403ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-10-01T02:30:11.414ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-10-01T02:30:11.414ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool)
322024-10-01T02:30:11.513ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal }
332024-10-01T02:30:11.921ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-10-01T02:30:11.921ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-10-01T02:30:11.921ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-10-01T02:30:11.921ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-10-01T02:30:11.921ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-10-01T02:30:11.921ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-10-01T02:30:11.921ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-10-01T02:30:11.930ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412024-10-01T02:30:11.930ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool)
422024-10-01T02:30:12.018ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal }
432024-10-01T02:30:12.426ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-10-01T02:30:12.426ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-10-01T02:30:12.426ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
462024-10-01T02:30:12.426ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
472024-10-01T02:30:12.426ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
482024-10-01T02:30:12.426ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
492024-10-01T02:30:12.426ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-10-01T02:30:12.426ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
512024-10-01T02:30:12.426ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
522024-10-01T02:30:12.426ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532024-10-01T02:30:12.426ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542024-10-01T02:30:12.426ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-10-01T02:30:12.426ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-10-01T02:30:12.426ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
572024-10-01T02:30:12.426ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
582024-10-01T02:30:12.426ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-10-01T02:30:12.426ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
602024-10-01T02:30:12.426ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-10-01T02:30:12.426ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-10-01T02:30:12.426ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
632024-10-01T02:30:12.426ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
642024-10-01T02:30:12.426ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
652024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
662024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-10-01T02:30:12.427ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-10-01T02:30:12.427ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-10-01T02:30:12.427ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-10-01T02:30:12.427ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-10-01T02:30:12.427ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-10-01T02:30:12.427ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
752024-10-01T02:30:12.427ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
762024-10-01T02:30:12.427ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
772024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
782024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-10-01T02:30:12.427ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-10-01T02:30:12.427ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
812024-10-01T02:30:12.427ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
822024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
832024-10-01T02:30:12.427ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
842024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-10-01T02:30:12.427ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-10-01T02:30:12.427ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-10-01T02:30:12.427ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-10-01T02:30:12.427ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-10-01T02:30:12.427ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-10-01T02:30:12.427ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-10-01T02:30:12.427ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-10-01T02:30:12.427ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-10-01T02:30:12.427ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-10-01T02:30:12.427ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
992024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1002024-10-01T02:30:12.427ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1012024-10-01T02:30:12.427ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
1022024-10-01T02:30:12.427ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1032024-10-01T02:30:12.427ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-10-01T02:30:12.427ZINFOSledAgent (bootstore): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-10-01T02:30:12.427ZINFOSledAgent (bootstore): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-10-01T02:30:12.427ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-10-01T02:30:12.427ZINFOSledAgent (bootstore): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-10-01T02:30:12.427ZINFOSledAgent (bootstore): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-10-01T02:30:12.427ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-10-01T02:30:12.427ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-10-01T02:30:12.427ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-10-01T02:30:12.437ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:737
1132024-10-01T02:30:12.437ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1665, tv_nsec: 280715597 } time_to_next_cleanup = 599.999936448s
1142024-10-01T02:30:12.437ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3915
1152024-10-01T02:30:12.437ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-10-01T02:30:12.440ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-10-01T02:30:12.440ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-10-01T02:30:12.440ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-10-01T02:30:12.442ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-10-01T02:30:12.442ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-10-01T02:30:12.442ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:72
1222024-10-01T02:30:12.442ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-10-01T02:30:12.442ZINFOSledAgent (RSS): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-10-01T02:30:12.442ZINFOSledAgent (RSS): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-10-01T02:30:12.442ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-10-01T02:30:12.442ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-10-01T02:30:12.442ZINFOSledAgent (RSS): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-10-01T02:30:12.442ZINFOSledAgent (RSS): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-10-01T02:30:12.442ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-10-01T02:30:12.442ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-10-01T02:30:12.442ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-10-01T02:30:12.442ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-10-01T02:30:12.442ZINFOSledAgent (RSS): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-10-01T02:30:12.443ZINFOSledAgent (RSS): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-10-01T02:30:12.443ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 52547763-7759-42e1-81a9-8b833f59f5dd, sleds: { [fdb0:18c0:4d0d:9fb2::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 2a571ec0-85a3-450e-88fc-680c13ffa211 (sled), rack_id: 52547763-7759-42e1-81a9-8b833f59f5dd, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-10-01T02:30:12.443ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-10-01T02:30:12.443ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-10-01T02:30:12.443ZINFOSledAgent (bootstore): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-10-01T02:30:12.443ZINFOSledAgent (bootstore): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-10-01T02:30:12.443ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-10-01T02:30:12.443ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 2a571ec0-85a3-450e-88fc-680c13ffa211 (sled), rack_id: 52547763-7759-42e1-81a9-8b833f59f5dd, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-10-01T02:30:12.444ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4212
1432024-10-01T02:30:12.444ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4594
1442024-10-01T02:30:12.449ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-10-01T02:30:12.454ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-10-01T02:30:12.454ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-10-01T02:30:12.459ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-10-01T02:30:12.459ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-10-01T02:30:12.459ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:40695
1502024-10-01T02:30:12.459ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 2a571ec0-85a3-450e-88fc-680c13ffa211 (sled), rack_id: 52547763-7759-42e1-81a9-8b833f59f5dd, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1512024-10-01T02:30:12.459ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1522024-10-01T02:30:12.459ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1532024-10-01T02:30:12.459ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:72
1542024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1552024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1562024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1572024-10-01T02:30:12.459ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1582024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-10-01T02:30:12.459ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-10-01T02:30:12.459ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-10-01T02:30:12.476ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-10-01T02:30:12.498ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-10-01T02:30:12.510ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1672024-10-01T02:30:12.510ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1682024-10-01T02:30:12.510ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1692024-10-01T02:30:12.510ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1702024-10-01T02:30:12.512ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 101.320745ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "eea86869-b4eb-49cd-b5f3-1b7e2e2b0156", "content-length": "124", "date": "Tue, 01 Oct 2024 02:30:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "eea86869-b4eb-49cd-b5f3-1b7e2e2b0156" } file = clients/ddm-admin-client/src/lib.rs:177
1712024-10-01T02:30:12.516ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1722024-10-01T02:30:12.523ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_1586f214-1a8d-4f2a-b9d3-907d2432a63f/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-10-01T02:30:12.546ZINFOSledAgent: successfully created encrypted zvol: name="oxi_1586f214-1a8d-4f2a-b9d3-907d2432a63f/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-10-01T02:30:12.546ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_1586f214-1a8d-4f2a-b9d3-907d2432a63f/swap" file = sled-agent/src/swap_device.rs:96
1752024-10-01T02:30:12.551ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1762024-10-01T02:30:12.551ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1772024-10-01T02:30:12.551ZINFOSledAgent (BackingFs): Ensuring dataset oxi_1586f214-1a8d-4f2a-b9d3-907d2432a63f/backing/fmd file = sled-agent/src/backing_fs.rs:138
1782024-10-01T02:30:12.615ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-10-01T02:30:12.615ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 155.68634ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d898f817-f446-4f36-b699-29715cb5e333", "content-length": "124", "date": "Tue, 01 Oct 2024 02:30:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d898f817-f446-4f36-b699-29715cb5e333" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-10-01T02:30:12.642ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1812024-10-01T02:30:12.772ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-10-01T02:30:12.773ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 138.618283ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "578d9fcb-b92c-43e8-80c5-481e0835dd15", "content-length": "124", "date": "Tue, 01 Oct 2024 02:30:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "578d9fcb-b92c-43e8-80c5-481e0835dd15" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-10-01T02:30:12.861ZINFOSledAgent (BackingFs): Mounting oxi_1586f214-1a8d-4f2a-b9d3-907d2432a63f/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1842024-10-01T02:30:12.874ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1852024-10-01T02:30:12.874ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1862024-10-01T02:30:12.874ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1872024-10-01T02:30:12.874ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1882024-10-01T02:30:12.913ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892024-10-01T02:30:12.913ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 244.941233ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c0ead987-7df5-4df8-a63f-7703c4928174", "content-length": "124", "date": "Tue, 01 Oct 2024 02:30:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c0ead987-7df5-4df8-a63f-7703c4928174" } file = clients/ddm-admin-client/src/lib.rs:177
1902024-10-01T02:30:13.160ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-10-01T02:30:13.160ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 217.972148ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5b7f8948-189c-4369-a629-06a7de5c7503", "content-length": "124", "date": "Tue, 01 Oct 2024 02:30:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5b7f8948-189c-4369-a629-06a7de5c7503" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-10-01T02:30:13.380ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-10-01T02:30:13.380ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 219.074197ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "98d6035b-3f73-4206-8c6e-6b91e25436b9", "content-length": "124", "date": "Tue, 01 Oct 2024 02:30:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "98d6035b-3f73-4206-8c6e-6b91e25436b9" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-10-01T02:30:13.587ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1952024-10-01T02:30:13.593ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:45292 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1962024-10-01T02:30:13.593ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:72 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1972024-10-01T02:30:13.593ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:45292 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1982024-10-01T02:30:13.594ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 234.417123ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
1992024-10-01T02:30:13.595ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2002024-10-01T02:30:13.595ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2012024-10-01T02:30:13.595ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:921 underlay_address = fd00:1122:3344:101::1
2022024-10-01T02:30:13.595ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:72
2032024-10-01T02:30:13.600ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2042024-10-01T02:30:13.830ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 602.316055ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2052024-10-01T02:30:14.434ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.232744735s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2062024-10-01T02:30:15.669ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.387802287s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2072024-10-01T02:30:16.073ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2670
2082024-10-01T02:30:16.073ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2888
2092024-10-01T02:30:16.073ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2540
2102024-10-01T02:30:16.073ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2587
2112024-10-01T02:30:16.073ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2969
2122024-10-01T02:30:16.073ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3011
2132024-10-01T02:30:16.075ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2813
2142024-10-01T02:30:16.076ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2581
2152024-10-01T02:30:16.076ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='4fd91d04-882e-49df-98b0-1430a0f50382'/> <propval type="astring" name="rack_id" value='52547763-7759-42e1-81a9-8b833f59f5dd'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:feb4:d6cf'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='2a571ec0-85a3-450e-88fc-680c13ffa211'/> <propval type="astring" name="rack_id" value='52547763-7759-42e1-81a9-8b833f59f5dd'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='2a571ec0-85a3-450e-88fc-680c13ffa211'/> <propval type="astring" name="rack_uuid" value='52547763-7759-42e1-81a9-8b833f59f5dd'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='2a571ec0-85a3-450e-88fc-680c13ffa211'/> <propval type="astring" name="rack_uuid" value='52547763-7759-42e1-81a9-8b833f59f5dd'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2162024-10-01T02:30:16.076ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2172024-10-01T02:30:16.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 54.303543ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2182024-10-01T02:30:16.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 116.212003ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-10-01T02:30:16.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 283.718601ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-10-01T02:30:17.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 480.476032ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-10-01T02:30:17.151ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.231911863s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2222024-10-01T02:30:17.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 434.807133ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-10-01T02:30:18.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 510.743821ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-10-01T02:30:18.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 805.379346ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-10-01T02:30:19.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.410102193s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-10-01T02:30:21.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.113525098s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-10-01T02:30:21.385ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.007485546s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2282024-10-01T02:30:22.728ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:12345
2292024-10-01T02:30:22.728ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2302024-10-01T02:30:22.728ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2312024-10-01T02:30:22.728ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2322024-10-01T02:30:22.728ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2332024-10-01T02:30:22.729ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2342024-10-01T02:30:22.729ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2352024-10-01T02:30:22.729ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2362024-10-01T02:30:22.729ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3915
2372024-10-01T02:30:22.969ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:40695
2382024-10-01T02:30:22.969ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2392024-10-01T02:30:22.969ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2402024-10-01T02:30:22.969ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-10-01T02:30:22.969ZINFOSledAgent (RSS): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2422024-10-01T02:30:22.969ZINFOSledAgent (RSS): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2432024-10-01T02:30:22.969ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-10-01T02:30:22.970ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-10-01T02:30:22.970ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-10-01T02:30:22.974ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4236 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2472024-10-01T02:30:22.974ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4252
2482024-10-01T02:30:22.974ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2492024-10-01T02:30:23.234ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54115
2502024-10-01T02:30:23.234ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2512024-10-01T02:30:23.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 325 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54115 req_id = f5d17b4f-8010-4be0-aed8-6c25c965f237 response_code = 200 uri = /inventory
2522024-10-01T02:30:23.264ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35558
2532024-10-01T02:30:23.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35558 req_id = 43f94ca6-b278-4899-bb9c-65fc117324a8 response_code = 200 uri = /sled-role
2542024-10-01T02:30:23.265ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2552024-10-01T02:30:23.265ZINFOSledAgent (RSS): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2562024-10-01T02:30:23.266ZINFOSledAgent (RSS): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2572024-10-01T02:30:23.266ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:1000
2582024-10-01T02:30:23.295ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 9ea4fbca-7825-4af5-b233-716faacf004a, pool_id: 52791279-0328-47f2-8641-652f421a0533 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 02b8c707-9e5c-4916-b5b0-7daa868b21fb, pool_id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: b3169590-463e-4aa5-8d74-46b767b4f418, pool_id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: c6ad05eb-5d45-4fe2-b244-fdb603095123, pool_id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: ed87e175-31f6-47cc-b01f-4d57cbb25b8e, pool_id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 783debfa-234e-46fa-9848-d866cf9fa69d, pool_id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: b34151d7-b6a2-4296-88e2-a5b8772c77c8, pool_id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: ae1124fc-68f6-4dfa-974f-6214f7f43390, pool_id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: fb87561f-21c8-407f-bf60-705375e63065, pool_id: 841bd0ff-a375-452e-9289-601e29937aad (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2592024-10-01T02:30:23.295ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61243
2602024-10-01T02:30:23.295ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:864 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2612024-10-01T02:30:23.295ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 9ea4fbca-7825-4af5-b233-716faacf004a, pool_id: 52791279-0328-47f2-8641-652f421a0533 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 02b8c707-9e5c-4916-b5b0-7daa868b21fb, pool_id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: b3169590-463e-4aa5-8d74-46b767b4f418, pool_id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: c6ad05eb-5d45-4fe2-b244-fdb603095123, pool_id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: ed87e175-31f6-47cc-b01f-4d57cbb25b8e, pool_id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 783debfa-234e-46fa-9848-d866cf9fa69d, pool_id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: b34151d7-b6a2-4296-88e2-a5b8772c77c8, pool_id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: ae1124fc-68f6-4dfa-974f-6214f7f43390, pool_id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: fb87561f-21c8-407f-bf60-705375e63065, pool_id: 841bd0ff-a375-452e-9289-601e29937aad (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2622024-10-01T02:30:23.295ZINFOSledAgent (StorageManager): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2632024-10-01T02:30:23.295ZINFOSledAgent (StorageManager): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2642024-10-01T02:30:23.295ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2652024-10-01T02:30:23.295ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2662024-10-01T02:30:23.295ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2672024-10-01T02:30:23.295ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2682024-10-01T02:30:23.295ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2692024-10-01T02:30:23.426ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2702024-10-01T02:30:23.426ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 52791279-0328-47f2-8641-652f421a0533 (zpool)
2712024-10-01T02:30:23.659ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2771459183353301701 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2722024-10-01T02:30:23.659ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12096287999618901980 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2732024-10-01T02:30:23.877ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2742024-10-01T02:30:23.882ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
2752024-10-01T02:30:23.938ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }
2762024-10-01T02:30:23.944ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2772024-10-01T02:30:23.945ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2782024-10-01T02:30:23.945ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2792024-10-01T02:30:23.945ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2802024-10-01T02:30:23.945ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2812024-10-01T02:30:23.945ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2822024-10-01T02:30:23.945ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2832024-10-01T02:30:23.977ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2842024-10-01T02:30:23.984ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_52791279-0328-47f2-8641-652f421a0533/crypt/zone file = sled-storage/src/dataset.rs:280
2852024-10-01T02:30:24.147ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_52791279-0328-47f2-8641-652f421a0533 file = sled-storage/src/dataset.rs:366
2862024-10-01T02:30:24.155ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2872024-10-01T02:30:24.155ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2882024-10-01T02:30:24.155ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2892024-10-01T02:30:24.155ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2902024-10-01T02:30:24.155ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2912024-10-01T02:30:24.165ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2922024-10-01T02:30:24.165ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool)
2932024-10-01T02:30:24.260ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }
2942024-10-01T02:30:24.268ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2952024-10-01T02:30:24.268ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
2962024-10-01T02:30:24.268ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
2972024-10-01T02:30:24.268ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
2982024-10-01T02:30:24.268ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2992024-10-01T02:30:24.268ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3002024-10-01T02:30:24.268ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3012024-10-01T02:30:24.303ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3022024-10-01T02:30:24.311ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_924badcc-07d3-4192-87c0-257ed1dc2017/crypt/zone file = sled-storage/src/dataset.rs:280
3032024-10-01T02:30:24.478ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_924badcc-07d3-4192-87c0-257ed1dc2017 file = sled-storage/src/dataset.rs:366
3042024-10-01T02:30:24.487ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3052024-10-01T02:30:24.487ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3062024-10-01T02:30:24.487ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3072024-10-01T02:30:24.487ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3082024-10-01T02:30:24.487ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3092024-10-01T02:30:24.497ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3102024-10-01T02:30:24.497ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool)
3112024-10-01T02:30:24.589ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }
3122024-10-01T02:30:24.598ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3132024-10-01T02:30:24.598ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3142024-10-01T02:30:24.598ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3152024-10-01T02:30:24.598ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3162024-10-01T02:30:24.598ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3172024-10-01T02:30:24.598ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3182024-10-01T02:30:24.598ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3192024-10-01T02:30:24.635ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3202024-10-01T02:30:24.643ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ff4b0e2e-bb96-457c-a345-9f8b3478a5a8/crypt/zone file = sled-storage/src/dataset.rs:280
3212024-10-01T02:30:24.729ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
3222024-10-01T02:30:24.817ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 file = sled-storage/src/dataset.rs:366
3232024-10-01T02:30:24.825ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3242024-10-01T02:30:24.825ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3252024-10-01T02:30:24.825ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3262024-10-01T02:30:24.825ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3272024-10-01T02:30:24.825ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3282024-10-01T02:30:24.834ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3292024-10-01T02:30:24.834ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool)
3302024-10-01T02:30:24.924ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }
3312024-10-01T02:30:24.931ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3322024-10-01T02:30:24.932ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3332024-10-01T02:30:24.932ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3342024-10-01T02:30:24.932ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3352024-10-01T02:30:24.932ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3362024-10-01T02:30:24.932ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3372024-10-01T02:30:24.932ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3382024-10-01T02:30:24.936ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
3392024-10-01T02:30:24.966ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3402024-10-01T02:30:24.974ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_efd83ea1-94bf-48aa-a059-ed387b79a77f/crypt/zone file = sled-storage/src/dataset.rs:280
3412024-10-01T02:30:25.142ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_efd83ea1-94bf-48aa-a059-ed387b79a77f file = sled-storage/src/dataset.rs:366
3422024-10-01T02:30:25.149ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3432024-10-01T02:30:25.149ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3442024-10-01T02:30:25.149ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3452024-10-01T02:30:25.149ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3462024-10-01T02:30:25.149ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3472024-10-01T02:30:25.159ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3482024-10-01T02:30:25.159ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool)
3492024-10-01T02:30:25.257ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }
3502024-10-01T02:30:25.266ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3512024-10-01T02:30:25.266ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3522024-10-01T02:30:25.266ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3532024-10-01T02:30:25.266ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3542024-10-01T02:30:25.266ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3552024-10-01T02:30:25.266ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3562024-10-01T02:30:25.266ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3572024-10-01T02:30:25.302ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3582024-10-01T02:30:25.310ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d/crypt/zone file = sled-storage/src/dataset.rs:280
3592024-10-01T02:30:25.396ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 20.959364908s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
3602024-10-01T02:30:25.489ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d file = sled-storage/src/dataset.rs:366
3612024-10-01T02:30:25.497ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3622024-10-01T02:30:25.497ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3632024-10-01T02:30:25.497ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3642024-10-01T02:30:25.497ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3652024-10-01T02:30:25.497ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3662024-10-01T02:30:25.507ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3672024-10-01T02:30:25.507ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool)
3682024-10-01T02:30:25.602ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool), kind: External }
3692024-10-01T02:30:25.611ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3702024-10-01T02:30:25.611ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3712024-10-01T02:30:25.611ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3722024-10-01T02:30:25.611ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3732024-10-01T02:30:25.611ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3742024-10-01T02:30:25.611ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3752024-10-01T02:30:25.611ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3762024-10-01T02:30:25.648ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3772024-10-01T02:30:25.656ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4b58fe92-3fb4-46b4-9ea6-d620062bf2a1/crypt/zone file = sled-storage/src/dataset.rs:280
3782024-10-01T02:30:25.832ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 file = sled-storage/src/dataset.rs:366
3792024-10-01T02:30:25.840ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3802024-10-01T02:30:25.840ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3812024-10-01T02:30:25.840ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3822024-10-01T02:30:25.840ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3832024-10-01T02:30:25.840ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3842024-10-01T02:30:25.850ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3852024-10-01T02:30:25.850ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool)
3862024-10-01T02:30:25.952ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool), kind: External }
3872024-10-01T02:30:25.960ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3882024-10-01T02:30:25.960ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3892024-10-01T02:30:25.960ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3902024-10-01T02:30:25.960ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3912024-10-01T02:30:25.960ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3922024-10-01T02:30:25.961ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3932024-10-01T02:30:25.961ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3942024-10-01T02:30:25.965ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
3952024-10-01T02:30:25.997ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3962024-10-01T02:30:26.005ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ee22443f-adfe-4fb0-bef1-d9b053b6a9d8/crypt/zone file = sled-storage/src/dataset.rs:280
3972024-10-01T02:30:26.178ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 file = sled-storage/src/dataset.rs:366
3982024-10-01T02:30:26.186ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3992024-10-01T02:30:26.186ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
4002024-10-01T02:30:26.186ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4012024-10-01T02:30:26.186ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4022024-10-01T02:30:26.186ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4032024-10-01T02:30:26.195ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4042024-10-01T02:30:26.195ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool)
4052024-10-01T02:30:26.286ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool), kind: External }
4062024-10-01T02:30:26.295ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4072024-10-01T02:30:26.295ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4082024-10-01T02:30:26.295ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4092024-10-01T02:30:26.295ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4102024-10-01T02:30:26.295ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4112024-10-01T02:30:26.295ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4122024-10-01T02:30:26.295ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4132024-10-01T02:30:26.329ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4142024-10-01T02:30:26.336ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a04e4a42-059d-4585-a24d-8ec28aa914e5/crypt/zone file = sled-storage/src/dataset.rs:280
4152024-10-01T02:30:26.517ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a04e4a42-059d-4585-a24d-8ec28aa914e5 file = sled-storage/src/dataset.rs:366
4162024-10-01T02:30:26.525ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4172024-10-01T02:30:26.525ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4182024-10-01T02:30:26.525ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4192024-10-01T02:30:26.525ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4202024-10-01T02:30:26.525ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4212024-10-01T02:30:26.535ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4222024-10-01T02:30:26.535ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 841bd0ff-a375-452e-9289-601e29937aad (zpool)
4232024-10-01T02:30:26.634ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 841bd0ff-a375-452e-9289-601e29937aad (zpool), kind: External }
4242024-10-01T02:30:26.643ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4252024-10-01T02:30:26.643ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4262024-10-01T02:30:26.643ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4272024-10-01T02:30:26.643ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4282024-10-01T02:30:26.644ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4292024-10-01T02:30:26.644ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4302024-10-01T02:30:26.644ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4312024-10-01T02:30:26.682ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4322024-10-01T02:30:26.690ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_841bd0ff-a375-452e-9289-601e29937aad/crypt/zone file = sled-storage/src/dataset.rs:280
4332024-10-01T02:30:26.729ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4342024-10-01T02:30:26.870ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_841bd0ff-a375-452e-9289-601e29937aad file = sled-storage/src/dataset.rs:366
4352024-10-01T02:30:26.878ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4362024-10-01T02:30:26.879ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4372024-10-01T02:30:26.879ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4382024-10-01T02:30:26.879ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1586f214-1a8d-4f2a-b9d3-907d2432a63f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09a98b58-1585-44a1-a9f9-fa7c634d7922 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 841bd0ff-a375-452e-9289-601e29937aad (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4392024-10-01T02:30:26.879ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4402024-10-01T02:30:26.879ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4412024-10-01T02:30:26.879ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4422024-10-01T02:30:26.879ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:868 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4432024-10-01T02:30:26.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4442024-10-01T02:30:26.879ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:894 generation = Generation(1) sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4452024-10-01T02:30:26.879ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:899 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4462024-10-01T02:30:26.879ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:904 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4472024-10-01T02:30:26.879ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4482024-10-01T02:30:26.879ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:910 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4492024-10-01T02:30:26.879ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:757 gen = Generation(1)
4502024-10-01T02:30:26.879ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:915 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4512024-10-01T02:30:26.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3584184 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:61243 req_id = 9e59cc68-6705-472e-84e9-a8f7504609e8 response_code = 200 uri = /omicron-physical-disks
4522024-10-01T02:30:26.911ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 293e78c2-b712-4c13-b502-e2be11a76306 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4532024-10-01T02:30:26.911ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33443
4542024-10-01T02:30:26.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4552024-10-01T02:30:26.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4562024-10-01T02:30:26.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4572024-10-01T02:30:26.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4582024-10-01T02:30:26.968ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4592024-10-01T02:30:27.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4602024-10-01T02:30:27.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4612024-10-01T02:30:27.064ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4622024-10-01T02:30:27.064ZINFOSledAgent (ServiceManager): No ledger in /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-zones.json file = common/src/ledger.rs:188
4632024-10-01T02:30:27.064ZINFOSledAgent (ServiceManager): No ledger in /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-zones.json file = common/src/ledger.rs:188
4642024-10-01T02:30:27.064ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4652024-10-01T02:30:27.109ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4662024-10-01T02:30:27.109ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4672024-10-01T02:30:27.109ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4682024-10-01T02:30:27.163ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306 file = illumos-utils/src/zone.rs:316
4692024-10-01T02:30:27.167ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef file = illumos-utils/src/zone.rs:316
4702024-10-01T02:30:27.171ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92 file = illumos-utils/src/zone.rs:316
4712024-10-01T02:30:27.183ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306 file = illumos-utils/src/zone.rs:354
4722024-10-01T02:30:27.188ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef file = illumos-utils/src/zone.rs:354
4732024-10-01T02:30:27.192ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92 file = illumos-utils/src/zone.rs:354
4742024-10-01T02:30:27.970ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
4752024-10-01T02:30:28.683ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4762024-10-01T02:30:28.683ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4772024-10-01T02:30:28.683ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
4782024-10-01T02:30:29.166ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4792024-10-01T02:30:29.645ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4802024-10-01T02:30:29.645ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4812024-10-01T02:30:29.645ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4822024-10-01T02:30:29.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 70.708764ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
4832024-10-01T02:30:29.655ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
4842024-10-01T02:30:29.659ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
4852024-10-01T02:30:29.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 108.443728ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
4862024-10-01T02:30:29.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 74.218783ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
4872024-10-01T02:30:29.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 40.079965ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
4882024-10-01T02:30:29.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 106.078871ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
4892024-10-01T02:30:30.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 244.359852ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
4902024-10-01T02:30:30.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 102.151995ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
4912024-10-01T02:30:30.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 121.667332ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
4922024-10-01T02:30:30.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 204.691471ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
4932024-10-01T02:30:30.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 353.869495ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
4942024-10-01T02:30:30.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 395.997309ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
4952024-10-01T02:30:30.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 485.99714ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
4962024-10-01T02:30:30.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 473.50884ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
4972024-10-01T02:30:30.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 1.001677886s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
4982024-10-01T02:30:30.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 843.36956ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
4992024-10-01T02:30:31.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 1.354881559s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
5002024-10-01T02:30:31.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 1.276916617s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
5012024-10-01T02:30:31.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 1.293130258s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
5022024-10-01T02:30:32.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 505.136318ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
5032024-10-01T02:30:32.531ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4260
5042024-10-01T02:30:32.531ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4270 gateway = fd00:1122:3344:101::1
5052024-10-01T02:30:32.536ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4349
5062024-10-01T02:30:32.661ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4397
5072024-10-01T02:30:32.661ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4424
5082024-10-01T02:30:32.700ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4437
5092024-10-01T02:30:32.700ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4301
5102024-10-01T02:30:32.733ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5155328440548043954 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
5112024-10-01T02:30:32.734ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16255590384494333248 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
5122024-10-01T02:30:32.751ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4343
5132024-10-01T02:30:32.751ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4404 rack_subnet = fd00:1122:3344::
5142024-10-01T02:30:32.772ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4415
5152024-10-01T02:30:32.772ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4457
5162024-10-01T02:30:32.871ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4489
5172024-10-01T02:30:32.871ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4495
5182024-10-01T02:30:32.895ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:55388
5192024-10-01T02:30:32.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:55388 req_id = 9b03831b-0196-4fa0-bcaa-210c5361b64f response_code = 200 uri = /sled-identifiers
5202024-10-01T02:30:32.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 585.136006ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
5212024-10-01T02:30:32.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 1.002530153s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
5222024-10-01T02:30:32.984ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4536
5232024-10-01T02:30:32.984ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5242024-10-01T02:30:33.017ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5252024-10-01T02:30:33.049ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5262024-10-01T02:30:33.049ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5272024-10-01T02:30:33.050ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 1
5282024-10-01T02:30:33.050ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5292024-10-01T02:30:33.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 624.947333ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
5302024-10-01T02:30:33.095ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
5312024-10-01T02:30:33.144ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5322024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5332024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5342024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5352024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5362024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5372024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5382024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5392024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5402024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5412024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5422024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5432024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5442024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5452024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5462024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5472024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5482024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5492024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5502024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5512024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5522024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5532024-10-01T02:30:33.144ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5542024-10-01T02:30:33.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306") failed: Property not found. retry in 986.248483ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
5552024-10-01T02:30:33.662ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2771459183353301701 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
5562024-10-01T02:30:33.662ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12096287999618901980 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
5572024-10-01T02:30:33.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 934.166192ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
5582024-10-01T02:30:33.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 607.989908ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
5592024-10-01T02:30:34.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92") failed: Property not found. retry in 739.246032ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
5602024-10-01T02:30:34.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef") failed: Property not found. retry in 763.794569ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
5612024-10-01T02:30:35.450ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_293e78c2-b712-4c13-b502-e2be11a76306
5622024-10-01T02:30:35.450ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_1af5fb25-d768-4bc6-915c-0b4c59e1f0ef
5632024-10-01T02:30:35.450ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_71ea2e4f-4772-4ce7-8093-48597fd7fd92
5642024-10-01T02:30:35.450ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5652024-10-01T02:30:35.450ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5662024-10-01T02:30:35.450ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8539510 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33443 req_id = 7ace7dfc-dd9f-4cc4-9850-174d82526f64 response_code = 204 uri = /omicron-zones
5672024-10-01T02:30:35.452ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5682024-10-01T02:30:35.499ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5692024-10-01T02:30:35.547ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5702024-10-01T02:30:35.595ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5712024-10-01T02:30:35.595ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5722024-10-01T02:30:35.629ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5732024-10-01T02:30:35.630ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5742024-10-01T02:30:35.630ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5752024-10-01T02:30:35.659ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 293e78c2-b712-4c13-b502-e2be11a76306 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4ffac138-75e0-477f-9e50-a3eaaace5f82 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 0f6137c5-e018-4955-81c4-5dfd48687c5a, kind: Service { id: 4ffac138-75e0-477f-9e50-a3eaaace5f82 }, name: Name("ntp-4ffac138-75e0-477f-9e50-a3eaaace5f82"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 219])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5762024-10-01T02:30:35.659ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55799
5772024-10-01T02:30:35.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5782024-10-01T02:30:35.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5792024-10-01T02:30:35.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5802024-10-01T02:30:35.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5812024-10-01T02:30:35.696ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5822024-10-01T02:30:35.696ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5832024-10-01T02:30:35.713ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5842024-10-01T02:30:35.713ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-zones.json file = common/src/ledger.rs:177
5852024-10-01T02:30:35.713ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-zones.json file = common/src/ledger.rs:177
5862024-10-01T02:30:35.714ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5872024-10-01T02:30:35.724ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5882024-10-01T02:30:35.756ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5892024-10-01T02:30:35.757ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5902024-10-01T02:30:35.757ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5912024-10-01T02:30:35.786ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 158, 219]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5922024-10-01T02:30:35.786ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = boundary_ntp
5932024-10-01T02:30:35.787ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5942024-10-01T02:30:35.787ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5952024-10-01T02:30:35.787ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5962024-10-01T02:30:35.787ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5972024-10-01T02:30:35.810ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82 file = illumos-utils/src/zone.rs:316
5982024-10-01T02:30:35.829ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82 file = illumos-utils/src/zone.rs:354
5992024-10-01T02:30:36.446ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6002024-10-01T02:30:36.446ZINFOSledAgent (ServiceManager): Booting oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6012024-10-01T02:30:36.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 64.792926ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6022024-10-01T02:30:36.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 59.029977ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6032024-10-01T02:30:36.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 272.818558ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6042024-10-01T02:30:37.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 558.922011ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6052024-10-01T02:30:37.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 527.270165ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6062024-10-01T02:30:38.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 1.456027585s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6072024-10-01T02:30:39.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 974.059673ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6082024-10-01T02:30:40.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82") failed: Property not found. retry in 1.307397341s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6092024-10-01T02:30:41.909ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82
6102024-10-01T02:30:41.909ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6112024-10-01T02:30:41.910ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6122024-10-01T02:30:41.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6250485 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55799 req_id = 2cecec7a-8f8f-4bd9-b139-922a0122af4a response_code = 204 uri = /omicron-zones
6132024-10-01T02:30:41.912ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6142024-10-01T02:30:41.942ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6152024-10-01T02:30:41.942ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47800
6162024-10-01T02:30:41.950ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3868
6172024-10-01T02:30:41.950ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47800 req_id = d1de9755-1e4d-492d-9264-dbc65bf39f68 response_code = 500 uri = /timesync
6182024-10-01T02:30:41.952ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6192024-10-01T02:30:42.059ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6202024-10-01T02:30:42.059ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57942
6212024-10-01T02:30:46.068ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3868
6222024-10-01T02:30:46.068ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4008677 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57942 req_id = a548d712-4235-4bac-adb2-65236b5557d0 response_code = 500 uri = /timesync
6232024-10-01T02:30:46.070ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2771459183353301701 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6242024-10-01T02:30:46.070ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12096287999618901980 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6252024-10-01T02:30:46.070ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15644254575437399705 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6262024-10-01T02:30:46.070ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3551025638267984902 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6272024-10-01T02:30:46.070ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15833821703518561678 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6282024-10-01T02:30:46.070ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5155328440548043954 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6292024-10-01T02:30:46.071ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16255590384494333248 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6302024-10-01T02:30:46.071ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6312024-10-01T02:30:46.222ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6322024-10-01T02:30:46.222ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61135
6332024-10-01T02:30:46.231ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3868
6342024-10-01T02:30:46.231ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8889 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61135 req_id = ed2e7cec-ab3f-4acd-b428-d066f6b4618f response_code = 500 uri = /timesync
6352024-10-01T02:30:46.233ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6362024-10-01T02:30:46.436ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6372024-10-01T02:30:46.436ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44711
6382024-10-01T02:30:46.447ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_4ffac138-75e0-477f-9e50-a3eaaace5f82': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3868
6392024-10-01T02:30:46.447ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 10871 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44711 req_id = 5c435205-46d8-4e85-8c19-9ddb1c2d9f9d response_code = 500 uri = /timesync
6402024-10-01T02:30:46.448ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6412024-10-01T02:30:46.733ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6422024-10-01T02:30:46.733ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61581
6432024-10-01T02:30:47.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1009076 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61581 req_id = d6a446d9-fa6e-4377-905f-d9d0de8e1162 response_code = 200 uri = /timesync
6442024-10-01T02:30:47.744ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6452024-10-01T02:30:47.744ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6462024-10-01T02:30:47.939ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6472024-10-01T02:30:47.939ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58078
6482024-10-01T02:30:47.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8473 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58078 req_id = 9431cbb2-820c-455e-bec0-6ee8090e8988 response_code = 200 uri = /timesync
6492024-10-01T02:30:47.949ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6502024-10-01T02:30:47.949ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6512024-10-01T02:30:48.269ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6522024-10-01T02:30:48.269ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39576
6532024-10-01T02:30:48.277ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8409 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39576 req_id = 6a438150-f5dc-433b-a0c4-e564dabfbe69 response_code = 200 uri = /timesync
6542024-10-01T02:30:48.280ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6552024-10-01T02:30:48.280ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6562024-10-01T02:30:48.231ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6572024-10-01T02:30:48.231ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43099
6582024-10-01T02:30:48.240ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8414 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43099 req_id = 67d0ca09-723e-476f-8cd7-fa65fbd87ad3 response_code = 200 uri = /timesync
6592024-10-01T02:30:48.241ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6602024-10-01T02:30:48.241ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6612024-10-01T02:30:48.256ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6622024-10-01T02:30:48.256ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35486
6632024-10-01T02:30:48.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9199 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35486 req_id = b3e2f274-cbe6-4493-9aeb-7776e073e5f9 response_code = 200 uri = /timesync
6642024-10-01T02:30:48.268ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6652024-10-01T02:30:48.268ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6662024-10-01T02:30:48.607ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6672024-10-01T02:30:48.608ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53135
6682024-10-01T02:30:48.616ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8321 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53135 req_id = 60564235-84bd-4bd6-9e14-25ffee064bc7 response_code = 200 uri = /timesync
6692024-10-01T02:30:48.618ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6702024-10-01T02:30:48.618ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6712024-10-01T02:30:49.269ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6722024-10-01T02:30:49.269ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39597
6732024-10-01T02:30:49.277ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7897 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39597 req_id = 824e8b6a-942d-46e6-9285-9a45a020a806 response_code = 200 uri = /timesync
6742024-10-01T02:30:49.279ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6752024-10-01T02:30:49.279ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6762024-10-01T02:30:49.711ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6772024-10-01T02:30:49.711ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38624
6782024-10-01T02:30:49.719ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8497 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38624 req_id = bc2c3a7a-65cb-4f67-a080-64fbbb2fdac9 response_code = 200 uri = /timesync
6792024-10-01T02:30:49.721ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.127256334 } file = sled-agent/src/rack_setup/service.rs:660
6802024-10-01T02:30:49.721ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6812024-10-01T02:30:50.548ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6822024-10-01T02:30:50.548ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44213
6832024-10-01T02:30:50.557ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8077 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44213 req_id = 16187f4a-d028-4fdb-bfc0-faa79f8d9a6b response_code = 200 uri = /timesync
6842024-10-01T02:30:50.558ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.12626639 } file = sled-agent/src/rack_setup/service.rs:660
6852024-10-01T02:30:50.558ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6862024-10-01T02:30:51.187ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 18235012463419925033 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6872024-10-01T02:30:51.187ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4101262359407211986 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
6882024-10-01T02:30:51.718ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6892024-10-01T02:30:51.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57376
6902024-10-01T02:30:51.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8538 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57376 req_id = 1a739653-0f79-4fae-b753-4c7619c9bb08 response_code = 200 uri = /timesync
6912024-10-01T02:30:51.729ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.123303965 } file = sled-agent/src/rack_setup/service.rs:660
6922024-10-01T02:30:51.729ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6932024-10-01T02:30:52.542ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6942024-10-01T02:30:52.542ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44296
6952024-10-01T02:30:52.552ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9468 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44296 req_id = ea703635-f52b-4976-8084-701338f82a67 response_code = 200 uri = /timesync
6962024-10-01T02:30:52.553ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.121006146 } file = sled-agent/src/rack_setup/service.rs:660
6972024-10-01T02:30:52.554ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6982024-10-01T02:30:53.505ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6992024-10-01T02:30:53.505ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44857
7002024-10-01T02:30:53.513ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7999 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44857 req_id = 3f41cc11-65ba-488d-b165-e64a0120f355 response_code = 200 uri = /timesync
7012024-10-01T02:30:53.516ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.118340455 } file = sled-agent/src/rack_setup/service.rs:660
7022024-10-01T02:30:53.516ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7032024-10-01T02:30:54.603ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7042024-10-01T02:30:54.603ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60490
7052024-10-01T02:30:54.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8202 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60490 req_id = 45a205da-bf71-481b-acc3-d5ce98562bd2 response_code = 200 uri = /timesync
7062024-10-01T02:30:54.613ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749853.947136, correction: 0.11537943 } file = sled-agent/src/rack_setup/service.rs:660
7072024-10-01T02:30:54.613ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7082024-10-01T02:30:55.357ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2771459183353301701 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7092024-10-01T02:30:55.357ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12096287999618901980 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7102024-10-01T02:30:55.357ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15644254575437399705 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7112024-10-01T02:30:55.357ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3551025638267984902 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7122024-10-01T02:30:55.357ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15833821703518561678 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7132024-10-01T02:30:55.357ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5155328440548043954 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7142024-10-01T02:30:55.357ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16255590384494333248 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7152024-10-01T02:30:56.519ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7162024-10-01T02:30:56.519ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64934
7172024-10-01T02:30:56.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8856 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64934 req_id = 0389c8db-3166-4c17-9df3-790e4fbb25b7 response_code = 200 uri = /timesync
7182024-10-01T02:30:56.532ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749855.2925007, correction: 0.109987892 } file = sled-agent/src/rack_setup/service.rs:660
7192024-10-01T02:30:56.532ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7202024-10-01T02:30:59.585ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7212024-10-01T02:30:59.585ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54377
7222024-10-01T02:30:59.593ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8342 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54377 req_id = e83a7406-8d1a-4b62-a8fa-4831d0c32101 response_code = 200 uri = /timesync
7232024-10-01T02:30:59.595ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749858.5084307, correction: 0.101785854 } file = sled-agent/src/rack_setup/service.rs:660
7242024-10-01T02:30:59.595ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7252024-10-01T02:31:00.694ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 250.345717ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7262024-10-01T02:31:01.215ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 18235012463419925033 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7272024-10-01T02:31:01.215ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4101262359407211986 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7282024-10-01T02:31:02.009ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35309
7292024-10-01T02:31:02.010ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 604 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35309 req_id = 2a669d80-58f5-4b74-889e-80e4a4f741b3 response_code = 200 uri = /baseboard
7302024-10-01T02:31:03.527ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7312024-10-01T02:31:03.527ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39263
7322024-10-01T02:31:03.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8598 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39263 req_id = d6bd6d73-4337-4de3-9dc9-663fdce6ab97 response_code = 200 uri = /timesync
7332024-10-01T02:31:03.540ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749862.285104, correction: 0.091094777 } file = sled-agent/src/rack_setup/service.rs:660
7342024-10-01T02:31:03.540ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7352024-10-01T02:31:05.386ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2771459183353301701 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7362024-10-01T02:31:05.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12096287999618901980 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7372024-10-01T02:31:05.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15644254575437399705 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7382024-10-01T02:31:05.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3551025638267984902 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7392024-10-01T02:31:05.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15833821703518561678 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7402024-10-01T02:31:05.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5155328440548043954 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7412024-10-01T02:31:05.387ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16255590384494333248 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7422024-10-01T02:31:07.974ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7432024-10-01T02:31:07.975ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35274
7442024-10-01T02:31:07.984ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8859 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35274 req_id = b66e5f90-0c7e-496b-a78b-76eba248dfa1 response_code = 200 uri = /timesync
7452024-10-01T02:31:07.987ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749866.375909, correction: 0.079101644 } file = sled-agent/src/rack_setup/service.rs:660
7462024-10-01T02:31:07.987ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7472024-10-01T02:31:11.246ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 18235012463419925033 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7482024-10-01T02:31:11.246ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4101262359407211986 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7492024-10-01T02:31:12.765ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7502024-10-01T02:31:12.765ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54794
7512024-10-01T02:31:12.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9183 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54794 req_id = 01d96c28-1068-4995-8e66-5e664061328a response_code = 200 uri = /timesync
7522024-10-01T02:31:12.777ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749872.2630672, correction: 0.066105045 } file = sled-agent/src/rack_setup/service.rs:660
7532024-10-01T02:31:12.777ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7542024-10-01T02:31:15.416ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2771459183353301701 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7552024-10-01T02:31:15.416ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12096287999618901980 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7562024-10-01T02:31:15.416ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15644254575437399705 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7572024-10-01T02:31:15.417ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 3551025638267984902 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7582024-10-01T02:31:15.417ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15833821703518561678 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7592024-10-01T02:31:15.417ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5155328440548043954 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7602024-10-01T02:31:15.417ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16255590384494333248 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7612024-10-01T02:31:15.951ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7622024-10-01T02:31:15.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63820
7632024-10-01T02:31:15.960ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63820 req_id = 6954d6b7-ae07-46a1-a866-2fade84d94a0 response_code = 200 uri = /timesync
7642024-10-01T02:31:15.965ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749874.5766137, correction: 0.057536893 } file = sled-agent/src/rack_setup/service.rs:660
7652024-10-01T02:31:15.965ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7662024-10-01T02:31:16.020ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 546.154933ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
7672024-10-01T02:31:20.485ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7682024-10-01T02:31:20.485ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56141
7692024-10-01T02:31:20.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22378 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56141 req_id = e7279676-56da-4fef-bedd-935afe715220 response_code = 200 uri = /timesync
7702024-10-01T02:31:20.512ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1727749879.4147227, correction: 0.045251463 } file = sled-agent/src/rack_setup/service.rs:660
7712024-10-01T02:31:20.512ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1299
7722024-10-01T02:31:20.542ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 293e78c2-b712-4c13-b502-e2be11a76306 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4ffac138-75e0-477f-9e50-a3eaaace5f82 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 0f6137c5-e018-4955-81c4-5dfd48687c5a, kind: Service { id: 4ffac138-75e0-477f-9e50-a3eaaace5f82 }, name: Name("ntp-4ffac138-75e0-477f-9e50-a3eaaace5f82"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 219])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 8ebcccef-2535-4e87-a7f6-0d24490b581d (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } } } }, OmicronZoneConfig { id: 37fee788-a460-4bbc-94b7-6cac019067a8 (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e5dd850-e7b4-444d-b7d0-264651e57d4a (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9005d6ef-ee79-47ee-a6b3-2b98a04766d6 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External } } } }, OmicronZoneConfig { id: e5b107ef-c7bd-453d-bcf6-3c79332eac2d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7732024-10-01T02:31:20.542ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48184
7742024-10-01T02:31:20.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7752024-10-01T02:31:20.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7762024-10-01T02:31:20.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7772024-10-01T02:31:20.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7782024-10-01T02:31:20.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7792024-10-01T02:31:20.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7802024-10-01T02:31:20.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ebcccef-2535-4e87-a7f6-0d24490b581d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7812024-10-01T02:31:20.605ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8ebcccef-2535-4e87-a7f6-0d24490b581d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7822024-10-01T02:31:20.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 37fee788-a460-4bbc-94b7-6cac019067a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7832024-10-01T02:31:20.662ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 37fee788-a460-4bbc-94b7-6cac019067a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7842024-10-01T02:31:20.716ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e5dd850-e7b4-444d-b7d0-264651e57d4a, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7852024-10-01T02:31:20.716ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4e5dd850-e7b4-444d-b7d0-264651e57d4a, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7862024-10-01T02:31:20.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9005d6ef-ee79-47ee-a6b3-2b98a04766d6, dataset_name: DatasetName { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7872024-10-01T02:31:20.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 9005d6ef-ee79-47ee-a6b3-2b98a04766d6, dataset_name: DatasetName { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7882024-10-01T02:31:20.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e5b107ef-c7bd-453d-bcf6-3c79332eac2d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7892024-10-01T02:31:20.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e5b107ef-c7bd-453d-bcf6-3c79332eac2d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }, kind: Cockroach }, responder: ... }
7902024-10-01T02:31:20.874ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7912024-10-01T02:31:20.874ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-zones.json file = common/src/ledger.rs:177
7922024-10-01T02:31:20.874ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-zones.json file = common/src/ledger.rs:177
7932024-10-01T02:31:20.874ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7942024-10-01T02:31:20.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7952024-10-01T02:31:20.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7962024-10-01T02:31:20.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7972024-10-01T02:31:20.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7982024-10-01T02:31:20.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7992024-10-01T02:31:21.048ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6 file = illumos-utils/src/zone.rs:316
8002024-10-01T02:31:21.052ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d file = illumos-utils/src/zone.rs:316
8012024-10-01T02:31:21.055ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d file = illumos-utils/src/zone.rs:316
8022024-10-01T02:31:21.060ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a file = illumos-utils/src/zone.rs:316
8032024-10-01T02:31:21.064ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8 file = illumos-utils/src/zone.rs:316
8042024-10-01T02:31:21.069ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6 file = illumos-utils/src/zone.rs:354
8052024-10-01T02:31:21.073ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d file = illumos-utils/src/zone.rs:354
8062024-10-01T02:31:21.078ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d file = illumos-utils/src/zone.rs:354
8072024-10-01T02:31:21.083ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a file = illumos-utils/src/zone.rs:354
8082024-10-01T02:31:21.087ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8 file = illumos-utils/src/zone.rs:354
8092024-10-01T02:31:24.046ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='9005d6ef-ee79-47ee-a6b3-2b98a04766d6'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8102024-10-01T02:31:24.047ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8112024-10-01T02:31:24.052ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e5b107ef-c7bd-453d-bcf6-3c79332eac2d'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8122024-10-01T02:31:24.053ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8132024-10-01T02:31:24.089ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='4e5dd850-e7b4-444d-b7d0-264651e57d4a'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8142024-10-01T02:31:24.089ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8152024-10-01T02:31:24.095ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='8ebcccef-2535-4e87-a7f6-0d24490b581d'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8162024-10-01T02:31:24.095ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8172024-10-01T02:31:24.102ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='37fee788-a460-4bbc-94b7-6cac019067a8'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8182024-10-01T02:31:24.103ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8192024-10-01T02:31:24.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 35.720703ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8202024-10-01T02:31:24.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 29.690053ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8212024-10-01T02:31:24.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 47.342558ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8222024-10-01T02:31:24.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 67.109092ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8232024-10-01T02:31:24.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 71.956578ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8242024-10-01T02:31:24.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 100.801721ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8252024-10-01T02:31:24.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 118.298849ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8262024-10-01T02:31:24.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 141.288192ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8272024-10-01T02:31:24.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 123.381254ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8282024-10-01T02:31:25.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 122.8127ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8292024-10-01T02:31:25.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 228.903582ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8302024-10-01T02:31:26.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 143.121956ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8312024-10-01T02:31:26.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 286.751178ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8322024-10-01T02:31:26.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 264.803837ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8332024-10-01T02:31:26.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 221.037732ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8342024-10-01T02:31:26.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 296.576679ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8352024-10-01T02:31:26.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 521.4944ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8362024-10-01T02:31:26.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 547.494254ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8372024-10-01T02:31:27.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 355.108161ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8382024-10-01T02:31:27.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 419.108643ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8392024-10-01T02:31:27.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 929.761222ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8402024-10-01T02:31:27.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 779.582969ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8412024-10-01T02:31:27.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 665.391282ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8422024-10-01T02:31:27.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 698.305552ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8432024-10-01T02:31:27.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 805.863937ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8442024-10-01T02:31:28.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 1.30640958s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8452024-10-01T02:31:28.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 1.444663156s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8462024-10-01T02:31:28.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.415166698s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8472024-10-01T02:31:28.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 807.933924ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8482024-10-01T02:31:28.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 1.129879383s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8492024-10-01T02:31:29.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 700.047631ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8502024-10-01T02:31:29.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 830.97554ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8512024-10-01T02:31:29.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 668.21396ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8522024-10-01T02:31:29.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 922.756561ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8532024-10-01T02:31:29.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.342445822s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8542024-10-01T02:31:29.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 748.952564ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8552024-10-01T02:31:30.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 673.045656ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8562024-10-01T02:31:30.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 643.667759ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8572024-10-01T02:31:30.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 1.411927446s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8582024-10-01T02:31:30.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 1.399837913s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8592024-10-01T02:31:30.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 1.372196359s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8602024-10-01T02:31:30.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 657.474623ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8612024-10-01T02:31:30.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.056307469s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8622024-10-01T02:31:31.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 587.217671ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8632024-10-01T02:31:31.645ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 719.797352ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
8642024-10-01T02:31:31.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 929.932498ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8652024-10-01T02:31:31.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 642.15739ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8662024-10-01T02:31:32.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 749.519782ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8672024-10-01T02:31:32.126ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59979
8682024-10-01T02:31:32.126ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59979 req_id = 9edb4460-dc3b-4b7f-8eed-6ce7cea54e33 response_code = 200 uri = /baseboard
8692024-10-01T02:31:32.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 589.429262ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8702024-10-01T02:31:32.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 1.163950728s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8712024-10-01T02:31:32.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 823.330501ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8722024-10-01T02:31:32.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 1.129565165s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8732024-10-01T02:31:32.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.231699994s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8742024-10-01T02:31:32.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 1.355081198s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8752024-10-01T02:31:33.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 902.70999ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8762024-10-01T02:31:33.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d") failed: Property not found. retry in 691.485218ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
8772024-10-01T02:31:33.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6") failed: Property not found. retry in 1.409231728s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
8782024-10-01T02:31:34.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.215076135s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8792024-10-01T02:31:34.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 1.419533951s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8802024-10-01T02:31:34.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 661.397253ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8812024-10-01T02:31:34.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 725.492536ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8822024-10-01T02:31:35.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 710.256603ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8832024-10-01T02:31:35.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 1.139228652s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8842024-10-01T02:31:35.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 1.487308336s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8852024-10-01T02:31:35.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 776.981049ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8862024-10-01T02:31:36.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.303422209s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8872024-10-01T02:31:36.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 728.828309ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8882024-10-01T02:31:37.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 1.121294629s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8892024-10-01T02:31:37.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 678.117945ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8902024-10-01T02:31:38.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 543.014724ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8912024-10-01T02:31:38.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 1.353740759s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8922024-10-01T02:31:38.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 1.34698882s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8932024-10-01T02:31:38.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 750.305044ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8942024-10-01T02:31:39.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.338114285s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8952024-10-01T02:31:39.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 997.805414ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8962024-10-01T02:31:39.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 1.029233423s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
8972024-10-01T02:31:40.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 764.354468ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
8982024-10-01T02:31:40.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.340196357s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
8992024-10-01T02:31:40.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 872.622533ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
9002024-10-01T02:31:41.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 1.076134384s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
9012024-10-01T02:31:41.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 1.255532779s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
9022024-10-01T02:31:42.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.151528586s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
9032024-10-01T02:31:42.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 600.633588ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
9042024-10-01T02:31:42.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 617.771354ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
9052024-10-01T02:31:43.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 902.757224ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
9062024-10-01T02:31:43.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d") failed: Property not found. retry in 1.402895133s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
9072024-10-01T02:31:43.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a") failed: Property not found. retry in 992.670708ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
9082024-10-01T02:31:43.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8") failed: Property not found. retry in 995.411619ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
9092024-10-01T02:31:45.007ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_9005d6ef-ee79-47ee-a6b3-2b98a04766d6
9102024-10-01T02:31:45.007ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_8ebcccef-2535-4e87-a7f6-0d24490b581d
9112024-10-01T02:31:45.007ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_e5b107ef-c7bd-453d-bcf6-3c79332eac2d
9122024-10-01T02:31:45.007ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_4e5dd850-e7b4-444d-b7d0-264651e57d4a
9132024-10-01T02:31:45.007ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_37fee788-a460-4bbc-94b7-6cac019067a8
9142024-10-01T02:31:45.007ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9152024-10-01T02:31:45.007ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9162024-10-01T02:31:45.008ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 24419687 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48184 req_id = 61e43c15-1bd3-42bf-a5d5-b2980d5e649a response_code = 204 uri = /omicron-zones
9172024-10-01T02:31:45.043ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52736
9182024-10-01T02:31:45.074ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::4]:32221 file = sled-agent/src/services.rs:3711
9192024-10-01T02:32:06.129ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3729
9202024-10-01T02:32:15.832ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3752
9212024-10-01T02:32:15.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 30788869 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:52736 req_id = df8f3d15-81ab-4463-94d0-6ed3a08408b9 response_code = 204 uri = /cockroachdb
9222024-10-01T02:32:15.834ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42032
9232024-10-01T02:32:15.835ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 338 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42032 req_id = 011790d7-f7d3-43c1-9022-79b68420d6af response_code = 200 uri = /baseboard
9242024-10-01T02:32:15.845ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.06670955s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
9252024-10-01T02:32:15.877ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 293e78c2-b712-4c13-b502-e2be11a76306 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 4ffac138-75e0-477f-9e50-a3eaaace5f82 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 0f6137c5-e018-4955-81c4-5dfd48687c5a, kind: Service { id: 4ffac138-75e0-477f-9e50-a3eaaace5f82 }, name: Name("ntp-4ffac138-75e0-477f-9e50-a3eaaace5f82"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 219])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 8ebcccef-2535-4e87-a7f6-0d24490b581d (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } } } }, OmicronZoneConfig { id: 37fee788-a460-4bbc-94b7-6cac019067a8 (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e5dd850-e7b4-444d-b7d0-264651e57d4a (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9005d6ef-ee79-47ee-a6b3-2b98a04766d6 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External } } } }, OmicronZoneConfig { id: e5b107ef-c7bd-453d-bcf6-3c79332eac2d (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External } } } }, OmicronZoneConfig { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 0a1ed896-5078-4267-b636-10151376f015 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e9e4c757-e076-4386-92e2-8217b8bb307d (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 841bd0ff-a375-452e-9289-601e29937aad (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 41f2943e-6041-44c9-a47a-3cf5fb012644 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 9df68fe6-52e1-420d-8e69-ae187fac2b5f (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5ee3adc6-bb70-4180-8a0e-1e58aba9f807 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: cdeda3ef-54c4-466c-bcd2-15530c779f17 (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 5ab93dca-3349-4e95-8adc-97fe1d9e5049 (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 44dd83be-d555-4f7e-91d1-342fa869f697 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0abff335-deb2-4991-b5b4-a3ef2f565fa9 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External } } } }, OmicronZoneConfig { id: c516b690-7599-4aaa-8432-aa06ded5383a (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External } } } }, OmicronZoneConfig { id: b4924e44-5dc5-45e0-a39b-c430868b3e63 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External } } } }, OmicronZoneConfig { id: 46e1cda7-578a-4867-bb73-b6c56c6231af (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External } } } }, OmicronZoneConfig { id: 3ae874d1-bf62-481b-aa5f-911ac7586252 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f46a1e4-9d58-499a-94fb-494a9375a290 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99df6194-3c68-48e6-a934-d49d19c5c7ee (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 841bd0ff-a375-452e-9289-601e29937aad (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 841bd0ff-a375-452e-9289-601e29937aad (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9262024-10-01T02:32:15.878ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44791
9272024-10-01T02:32:15.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9282024-10-01T02:32:15.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 293e78c2-b712-4c13-b502-e2be11a76306, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9292024-10-01T02:32:15.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9302024-10-01T02:32:15.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1af5fb25-d768-4bc6-915c-0b4c59e1f0ef, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9312024-10-01T02:32:15.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9322024-10-01T02:32:15.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ea2e4f-4772-4ce7-8093-48597fd7fd92, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9332024-10-01T02:32:15.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ebcccef-2535-4e87-a7f6-0d24490b581d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9342024-10-01T02:32:15.948ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8ebcccef-2535-4e87-a7f6-0d24490b581d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9352024-10-01T02:32:15.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 37fee788-a460-4bbc-94b7-6cac019067a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9362024-10-01T02:32:15.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 37fee788-a460-4bbc-94b7-6cac019067a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9372024-10-01T02:32:15.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e5dd850-e7b4-444d-b7d0-264651e57d4a, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9382024-10-01T02:32:15.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4e5dd850-e7b4-444d-b7d0-264651e57d4a, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9392024-10-01T02:32:16.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9005d6ef-ee79-47ee-a6b3-2b98a04766d6, dataset_name: DatasetName { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9402024-10-01T02:32:16.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 9005d6ef-ee79-47ee-a6b3-2b98a04766d6, dataset_name: DatasetName { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9412024-10-01T02:32:16.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e5b107ef-c7bd-453d-bcf6-3c79332eac2d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9422024-10-01T02:32:16.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e5b107ef-c7bd-453d-bcf6-3c79332eac2d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }, kind: Cockroach }, responder: ... }
9432024-10-01T02:32:16.046ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eb937866-c779-4a9e-aec4-cc18302d4ce1, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9442024-10-01T02:32:16.046ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: eb937866-c779-4a9e-aec4-cc18302d4ce1, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9452024-10-01T02:32:16.139ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a1ed896-5078-4267-b636-10151376f015, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9462024-10-01T02:32:16.139ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0a1ed896-5078-4267-b636-10151376f015, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9472024-10-01T02:32:16.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9df68fe6-52e1-420d-8e69-ae187fac2b5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
9482024-10-01T02:32:16.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 9df68fe6-52e1-420d-8e69-ae187fac2b5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9492024-10-01T02:32:16.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 44dd83be-d555-4f7e-91d1-342fa869f697, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9502024-10-01T02:32:16.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 44dd83be-d555-4f7e-91d1-342fa869f697, dataset_name: DatasetName { pool_name: ZpoolName { id: 52791279-0328-47f2-8641-652f421a0533 (zpool), kind: External }, kind: Crucible }, responder: ... }
9512024-10-01T02:32:16.442ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0abff335-deb2-4991-b5b4-a3ef2f565fa9, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9522024-10-01T02:32:16.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0abff335-deb2-4991-b5b4-a3ef2f565fa9, dataset_name: DatasetName { pool_name: ZpoolName { id: 924badcc-07d3-4192-87c0-257ed1dc2017 (zpool), kind: External }, kind: Crucible }, responder: ... }
9532024-10-01T02:32:16.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c516b690-7599-4aaa-8432-aa06ded5383a, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9542024-10-01T02:32:16.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c516b690-7599-4aaa-8432-aa06ded5383a, dataset_name: DatasetName { pool_name: ZpoolName { id: ff4b0e2e-bb96-457c-a345-9f8b3478a5a8 (zpool), kind: External }, kind: Crucible }, responder: ... }
9552024-10-01T02:32:16.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b4924e44-5dc5-45e0-a39b-c430868b3e63, dataset_name: DatasetName { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9562024-10-01T02:32:16.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b4924e44-5dc5-45e0-a39b-c430868b3e63, dataset_name: DatasetName { pool_name: ZpoolName { id: efd83ea1-94bf-48aa-a059-ed387b79a77f (zpool), kind: External }, kind: Crucible }, responder: ... }
9572024-10-01T02:32:16.715ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46e1cda7-578a-4867-bb73-b6c56c6231af, dataset_name: DatasetName { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9582024-10-01T02:32:16.715ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 46e1cda7-578a-4867-bb73-b6c56c6231af, dataset_name: DatasetName { pool_name: ZpoolName { id: a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d (zpool), kind: External }, kind: Crucible }, responder: ... }
9592024-10-01T02:32:16.807ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3ae874d1-bf62-481b-aa5f-911ac7586252, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9602024-10-01T02:32:16.807ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3ae874d1-bf62-481b-aa5f-911ac7586252, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b58fe92-3fb4-46b4-9ea6-d620062bf2a1 (zpool), kind: External }, kind: Crucible }, responder: ... }
9612024-10-01T02:32:16.878ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a, dataset_name: DatasetName { pool_name: ZpoolName { id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9622024-10-01T02:32:16.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a, dataset_name: DatasetName { pool_name: ZpoolName { id: ee22443f-adfe-4fb0-bef1-d9b053b6a9d8 (zpool), kind: External }, kind: Crucible }, responder: ... }
9632024-10-01T02:32:16.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f46a1e4-9d58-499a-94fb-494a9375a290, dataset_name: DatasetName { pool_name: ZpoolName { id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9642024-10-01T02:32:16.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7f46a1e4-9d58-499a-94fb-494a9375a290, dataset_name: DatasetName { pool_name: ZpoolName { id: a04e4a42-059d-4585-a24d-8ec28aa914e5 (zpool), kind: External }, kind: Crucible }, responder: ... }
9652024-10-01T02:32:17.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99df6194-3c68-48e6-a934-d49d19c5c7ee, dataset_name: DatasetName { pool_name: ZpoolName { id: 841bd0ff-a375-452e-9289-601e29937aad (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9662024-10-01T02:32:17.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99df6194-3c68-48e6-a934-d49d19c5c7ee, dataset_name: DatasetName { pool_name: ZpoolName { id: 841bd0ff-a375-452e-9289-601e29937aad (zpool), kind: External }, kind: Crucible }, responder: ... }
9672024-10-01T02:32:17.135ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9682024-10-01T02:32:17.135ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-zones.json file = common/src/ledger.rs:177
9692024-10-01T02:32:17.135ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-zones.json file = common/src/ledger.rs:177
9702024-10-01T02:32:17.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9712024-10-01T02:32:17.274ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9722024-10-01T02:32:17.275ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9732024-10-01T02:32:17.302ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9742024-10-01T02:32:17.302ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9752024-10-01T02:32:17.302ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9762024-10-01T02:32:17.302ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9772024-10-01T02:32:17.329ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9782024-10-01T02:32:17.329ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9792024-10-01T02:32:17.329ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9802024-10-01T02:32:17.329ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9812024-10-01T02:32:17.494ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9822024-10-01T02:32:17.566ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9832024-10-01T02:32:17.566ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9842024-10-01T02:32:17.566ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9852024-10-01T02:32:17.566ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9862024-10-01T02:32:17.566ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9872024-10-01T02:32:17.566ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9882024-10-01T02:32:17.566ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9892024-10-01T02:32:17.567ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
9902024-10-01T02:32:17.568ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f file = illumos-utils/src/zone.rs:316
9912024-10-01T02:32:17.573ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644 file = illumos-utils/src/zone.rs:316
9922024-10-01T02:32:17.577ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af file = illumos-utils/src/zone.rs:316
9932024-10-01T02:32:17.582ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a file = illumos-utils/src/zone.rs:316
9942024-10-01T02:32:17.588ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697 file = illumos-utils/src/zone.rs:316
9952024-10-01T02:32:17.593ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252 file = illumos-utils/src/zone.rs:316
9962024-10-01T02:32:17.599ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807 file = illumos-utils/src/zone.rs:316
9972024-10-01T02:32:17.604ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17 file = illumos-utils/src/zone.rs:316
9982024-10-01T02:32:17.729ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f file = illumos-utils/src/zone.rs:354
9992024-10-01T02:32:17.734ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644 file = illumos-utils/src/zone.rs:354
10002024-10-01T02:32:17.739ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af file = illumos-utils/src/zone.rs:354
10012024-10-01T02:32:17.744ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a file = illumos-utils/src/zone.rs:354
10022024-10-01T02:32:17.748ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697 file = illumos-utils/src/zone.rs:354
10032024-10-01T02:32:17.753ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252 file = illumos-utils/src/zone.rs:354
10042024-10-01T02:32:17.758ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807 file = illumos-utils/src/zone.rs:354
10052024-10-01T02:32:17.763ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17 file = illumos-utils/src/zone.rs:354
10062024-10-01T02:32:17.768ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049 file = illumos-utils/src/zone.rs:316
10072024-10-01T02:32:17.774ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9 file = illumos-utils/src/zone.rs:316
10082024-10-01T02:32:17.779ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a file = illumos-utils/src/zone.rs:316
10092024-10-01T02:32:17.785ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63 file = illumos-utils/src/zone.rs:316
10102024-10-01T02:32:17.791ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290 file = illumos-utils/src/zone.rs:316
10112024-10-01T02:32:17.796ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee file = illumos-utils/src/zone.rs:316
10122024-10-01T02:32:17.842ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10132024-10-01T02:32:17.843ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049 file = illumos-utils/src/zone.rs:354
10142024-10-01T02:32:17.848ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9 file = illumos-utils/src/zone.rs:354
10152024-10-01T02:32:17.853ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a file = illumos-utils/src/zone.rs:354
10162024-10-01T02:32:17.858ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63 file = illumos-utils/src/zone.rs:354
10172024-10-01T02:32:17.863ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290 file = illumos-utils/src/zone.rs:354
10182024-10-01T02:32:17.868ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee file = illumos-utils/src/zone.rs:354
10192024-10-01T02:32:17.911ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10202024-10-01T02:32:17.957ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10212024-10-01T02:32:17.994ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10222024-10-01T02:32:17.996ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10232024-10-01T02:32:17.996ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10242024-10-01T02:32:18.044ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 254, 225]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
10252024-10-01T02:32:18.044ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = external_dns
10262024-10-01T02:32:18.106ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
10272024-10-01T02:32:18.108ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10282024-10-01T02:32:18.108ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10292024-10-01T02:32:18.171ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 139, 222]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10302024-10-01T02:32:18.171ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
10312024-10-01T02:32:18.173ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10322024-10-01T02:32:18.173ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10332024-10-01T02:32:18.232ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 226, 149]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
10342024-10-01T02:32:18.232ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = external_dns
10352024-10-01T02:32:18.234ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10362024-10-01T02:32:18.234ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10372024-10-01T02:32:18.284ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 167, 29]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10382024-10-01T02:32:18.284ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
10392024-10-01T02:32:18.284ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
10402024-10-01T02:32:18.284ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
10412024-10-01T02:32:18.286ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
10422024-10-01T02:32:18.286ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
10432024-10-01T02:32:18.336ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 176, 241]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10442024-10-01T02:32:18.336ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
10452024-10-01T02:32:18.336ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
10462024-10-01T02:32:18.338ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
10472024-10-01T02:32:18.338ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
10482024-10-01T02:32:18.338ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10492024-10-01T02:32:18.338ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
10502024-10-01T02:32:18.338ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
10512024-10-01T02:32:18.376ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
10522024-10-01T02:32:18.380ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
10532024-10-01T02:32:18.380ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
10542024-10-01T02:32:18.380ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10552024-10-01T02:32:18.380ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
10562024-10-01T02:32:18.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10572024-10-01T02:32:18.439ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
10582024-10-01T02:32:18.512ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10592024-10-01T02:32:18.512ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
10602024-10-01T02:32:18.512ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
10612024-10-01T02:32:18.512ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1 file = illumos-utils/src/zone.rs:316
10622024-10-01T02:32:18.538ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6 file = illumos-utils/src/zone.rs:316
10632024-10-01T02:32:18.607ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015 file = illumos-utils/src/zone.rs:316
10642024-10-01T02:32:18.615ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
10652024-10-01T02:32:18.619ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10662024-10-01T02:32:18.619ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1 file = illumos-utils/src/zone.rs:354
10672024-10-01T02:32:18.627ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6 file = illumos-utils/src/zone.rs:354
10682024-10-01T02:32:18.636ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 file = illumos-utils/src/zone.rs:316
10692024-10-01T02:32:18.734ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015 file = illumos-utils/src/zone.rs:354
10702024-10-01T02:32:18.765ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d file = illumos-utils/src/zone.rs:316
10712024-10-01T02:32:18.793ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 file = illumos-utils/src/zone.rs:354
10722024-10-01T02:32:18.862ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d file = illumos-utils/src/zone.rs:354
10732024-10-01T02:32:20.757ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10742024-10-01T02:32:20.757ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
10752024-10-01T02:32:20.764ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10762024-10-01T02:32:20.765ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10772024-10-01T02:32:20.765ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
10782024-10-01T02:32:20.771ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
10792024-10-01T02:32:20.866ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5ff15611-7292-4caf-bb48-0581e1a67473'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10802024-10-01T02:32:20.866ZINFOSledAgent (ServiceManager): Booting oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
10812024-10-01T02:32:20.875ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_ee22443f-adfe-4fb0-bef1-d9b053b6a9d8/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='18edb4f3-ff9d-41ec-9ab1-0b2748e9296f'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10822024-10-01T02:32:20.875ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_52791279-0328-47f2-8641-652f421a0533/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='341301bf-0429-402d-a32d-a4604e12a1e3'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10832024-10-01T02:32:20.877ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_4b58fe92-3fb4-46b4-9ea6-d620062bf2a1/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='03af95ab-c6b7-4e00-b98f-5d4d709da716'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10842024-10-01T02:32:20.877ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_924badcc-07d3-4192-87c0-257ed1dc2017/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='f1eacf36-34a5-40fa-b994-3432ebdd771c'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10852024-10-01T02:32:20.877ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_efd83ea1-94bf-48aa-a059-ed387b79a77f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3f31d9be-7c18-4de3-913e-aba8db87719c'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10862024-10-01T02:32:20.877ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a04e4a42-059d-4585-a24d-8ec28aa914e5/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='75a4b3fc-d403-4ece-aefe-6950188537b7'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10872024-10-01T02:32:20.877ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_841bd0ff-a375-452e-9289-601e29937aad/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='765773a0-b2d7-4425-b94e-6bbce719cae9'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10882024-10-01T02:32:20.877ZINFOSledAgent (ServiceManager): Booting oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
10892024-10-01T02:32:20.885ZINFOSledAgent (ServiceManager): Booting oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
10902024-10-01T02:32:20.892ZINFOSledAgent (ServiceManager): Booting oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
10912024-10-01T02:32:20.899ZINFOSledAgent (ServiceManager): Booting oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
10922024-10-01T02:32:20.906ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_ff4b0e2e-bb96-457c-a345-9f8b3478a5a8/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='737e48fc-49d3-4dfe-861b-dad1ea05158f'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10932024-10-01T02:32:20.906ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
10942024-10-01T02:32:20.913ZINFOSledAgent (ServiceManager): Booting oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
10952024-10-01T02:32:20.920ZINFOSledAgent (ServiceManager): Booting oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
10962024-10-01T02:32:20.927ZINFOSledAgent (ServiceManager): Booting oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
10972024-10-01T02:32:20.996ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10982024-10-01T02:32:20.996ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
10992024-10-01T02:32:21.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 29.611211ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
11002024-10-01T02:32:21.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 30.865106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
11012024-10-01T02:32:21.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 50.755085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
11022024-10-01T02:32:21.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 112.518478ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
11032024-10-01T02:32:21.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 88.594581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
11042024-10-01T02:32:21.103ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='41f2943e-6041-44c9-a47a-3cf5fb012644'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11052024-10-01T02:32:21.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 54.439019ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
11062024-10-01T02:32:21.121ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
11072024-10-01T02:32:21.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 73.94739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
11082024-10-01T02:32:21.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 292.920425ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
11092024-10-01T02:32:21.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 110.677338ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
11102024-10-01T02:32:27.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 173.874383ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
11112024-10-01T02:32:28.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 46.718628ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
11122024-10-01T02:32:28.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 25.442312ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
11132024-10-01T02:32:28.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 41.364408ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
11142024-10-01T02:32:28.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 65.483566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
11152024-10-01T02:32:28.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 67.054624ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
11162024-10-01T02:32:28.391ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11172024-10-01T02:32:28.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 56.286583ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
11182024-10-01T02:32:28.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 252.538041ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
11192024-10-01T02:32:28.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 52.886702ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
11202024-10-01T02:32:28.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 49.73408ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
11212024-10-01T02:32:28.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 73.322715ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
11222024-10-01T02:32:28.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 217.295133ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
11232024-10-01T02:32:28.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 203.002487ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
11242024-10-01T02:32:28.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 582.310249ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
11252024-10-01T02:32:28.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 120.018243ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
11262024-10-01T02:32:28.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 81.442848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
11272024-10-01T02:32:28.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 69.670951ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
11282024-10-01T02:32:28.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 94.068578ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
11292024-10-01T02:32:28.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 113.972217ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
11302024-10-01T02:32:28.824ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
11312024-10-01T02:32:28.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 139.872455ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
11322024-10-01T02:32:28.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 61.676758ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
11332024-10-01T02:32:28.874ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 372.224242ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
11342024-10-01T02:32:28.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 58.977122ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
11352024-10-01T02:32:28.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 76.670934ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
11362024-10-01T02:32:28.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 127.618086ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
11372024-10-01T02:32:28.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 947.09468ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
11382024-10-01T02:32:28.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 529.631772ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
11392024-10-01T02:32:28.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 275.245079ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
11402024-10-01T02:32:28.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 160.20957ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
11412024-10-01T02:32:28.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 278.995693ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
11422024-10-01T02:32:28.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 204.37063ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
11432024-10-01T02:32:29.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 231.247188ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
11442024-10-01T02:32:29.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 281.316593ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
11452024-10-01T02:32:29.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 110.628079ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
11462024-10-01T02:32:29.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 277.455669ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
11472024-10-01T02:32:29.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 230.887731ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
11482024-10-01T02:32:29.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 236.236921ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
11492024-10-01T02:32:29.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 41.960827ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
11502024-10-01T02:32:29.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 439.063952ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
11512024-10-01T02:32:29.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 251.982939ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
11522024-10-01T02:32:29.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 75.638166ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
11532024-10-01T02:32:29.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 945.653141ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
11542024-10-01T02:32:29.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 351.398165ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
11552024-10-01T02:32:29.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 493.925404ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
11562024-10-01T02:32:29.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 425.496891ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
11572024-10-01T02:32:29.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 281.233676ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
11582024-10-01T02:32:29.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 601.130171ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
11592024-10-01T02:32:29.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 209.154212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
11602024-10-01T02:32:29.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 365.515927ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
11612024-10-01T02:32:29.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 294.10086ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
11622024-10-01T02:32:29.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 521.764367ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
11632024-10-01T02:32:29.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 488.368596ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
11642024-10-01T02:32:29.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 526.244755ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
11652024-10-01T02:32:29.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 720.133713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
11662024-10-01T02:32:29.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 420.134272ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
11672024-10-01T02:32:29.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 251.624113ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
11682024-10-01T02:32:29.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 639.75279ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
11692024-10-01T02:32:29.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 656.091708ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
11702024-10-01T02:32:29.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 684.477544ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
11712024-10-01T02:32:29.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 961.053912ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
11722024-10-01T02:32:29.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 767.280519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
11732024-10-01T02:32:29.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 858.043684ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
11742024-10-01T02:32:29.825ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11752024-10-01T02:32:29.825ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11762024-10-01T02:32:29.825ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11772024-10-01T02:32:29.825ZINFOSledAgent (ServiceManager): Booting oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
11782024-10-01T02:32:29.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 613.054585ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
11792024-10-01T02:32:29.848ZINFOSledAgent (ServiceManager): Booting oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
11802024-10-01T02:32:29.855ZINFOSledAgent (ServiceManager): Booting oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
11812024-10-01T02:32:29.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 887.863978ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
11822024-10-01T02:32:29.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 534.995173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
11832024-10-01T02:32:29.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 1.48444299s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
11842024-10-01T02:32:29.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 691.624939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
11852024-10-01T02:32:29.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 911.594089ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
11862024-10-01T02:32:29.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 943.375107ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
11872024-10-01T02:32:30.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 40.481291ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
11882024-10-01T02:32:30.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 59.320574ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
11892024-10-01T02:32:30.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 120.697916ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
11902024-10-01T02:32:30.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 66.279312ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
11912024-10-01T02:32:30.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 136.924416ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
11922024-10-01T02:32:30.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 1.424826788s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
11932024-10-01T02:32:30.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 102.979197ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
11942024-10-01T02:32:30.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 1.080280937s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
11952024-10-01T02:32:30.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 1.191677098s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
11962024-10-01T02:32:30.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 223.161553ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
11972024-10-01T02:32:30.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 589.329701ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
11982024-10-01T02:32:30.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 127.417669ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
11992024-10-01T02:32:30.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 231.548806ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
12002024-10-01T02:32:30.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 1.367092749s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
12012024-10-01T02:32:30.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.2616039s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
12022024-10-01T02:32:30.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 441.924758ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
12032024-10-01T02:32:30.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 1.207195577s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
12042024-10-01T02:32:30.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 261.164548ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
12052024-10-01T02:32:30.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 664.079489ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
12062024-10-01T02:32:30.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 548.33564ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
12072024-10-01T02:32:30.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 801.265161ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
12082024-10-01T02:32:30.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 1.34439203s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
12092024-10-01T02:32:30.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.113445655s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
12102024-10-01T02:32:30.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 511.897198ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
12112024-10-01T02:32:30.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 1.418618883s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
12122024-10-01T02:32:30.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 1.438722379s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
12132024-10-01T02:32:30.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 1.316055407s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
12142024-10-01T02:32:30.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 1.175381925s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
12152024-10-01T02:32:30.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.351612699s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
12162024-10-01T02:32:31.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 738.382159ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
12172024-10-01T02:32:31.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.14148648s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
12182024-10-01T02:32:31.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 776.176007ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
12192024-10-01T02:32:31.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 521.118387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
12202024-10-01T02:32:31.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 570.448268ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
12212024-10-01T02:32:31.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 1.221775041s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
12222024-10-01T02:32:31.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 569.45764ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
12232024-10-01T02:32:31.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 1.252421007s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
12242024-10-01T02:32:31.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 512.773621ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
12252024-10-01T02:32:31.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.013092758s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
12262024-10-01T02:32:31.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 1.25497623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
12272024-10-01T02:32:31.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.379689008s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
12282024-10-01T02:32:31.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 967.838527ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
12292024-10-01T02:32:32.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 909.471461ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
12302024-10-01T02:32:32.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 1.02508586s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
12312024-10-01T02:32:32.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 984.318125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
12322024-10-01T02:32:32.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 669.908614ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
12332024-10-01T02:32:32.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 881.594693ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
12342024-10-01T02:32:32.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 1.267730324s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
12352024-10-01T02:32:32.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 721.930081ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
12362024-10-01T02:32:32.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 1.155536778s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
12372024-10-01T02:32:32.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 1.279853799s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
12382024-10-01T02:32:32.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 772.788762ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
12392024-10-01T02:32:32.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 993.012563ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
12402024-10-01T02:32:32.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.076139956s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
12412024-10-01T02:32:32.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 907.485063ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
12422024-10-01T02:32:32.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.011386579s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
12432024-10-01T02:32:32.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 981.340731ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
12442024-10-01T02:32:32.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 1.459863341s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
12452024-10-01T02:32:32.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 928.078175ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
12462024-10-01T02:32:32.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 705.025645ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
12472024-10-01T02:32:32.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 542.819391ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
12482024-10-01T02:32:33.000ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.509770095s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
12492024-10-01T02:32:33.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 1.464357918s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
12502024-10-01T02:32:33.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 795.585135ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
12512024-10-01T02:32:33.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 582.881852ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
12522024-10-01T02:32:33.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 1.07846294s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
12532024-10-01T02:32:33.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 903.981233ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
12542024-10-01T02:32:33.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.353332602s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
12552024-10-01T02:32:33.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 651.185308ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
12562024-10-01T02:32:33.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 1.288062551s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
12572024-10-01T02:32:33.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.088594091s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
12582024-10-01T02:32:33.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 1.097025968s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
12592024-10-01T02:32:33.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 589.129147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
12602024-10-01T02:32:33.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 555.415197ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
12612024-10-01T02:32:33.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 1.49761078s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
12622024-10-01T02:32:33.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.301335729s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
12632024-10-01T02:32:33.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 526.562553ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
12642024-10-01T02:32:33.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 731.24278ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
12652024-10-01T02:32:33.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 1.37923373s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
12662024-10-01T02:32:33.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 1.331643541s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
12672024-10-01T02:32:33.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 1.414911899s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
12682024-10-01T02:32:34.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 1.301745123s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
12692024-10-01T02:32:34.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 849.902528ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
12702024-10-01T02:32:34.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 1.348418629s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
12712024-10-01T02:32:34.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 515.382973ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
12722024-10-01T02:32:34.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 770.600799ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
12732024-10-01T02:32:34.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.287271235s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
12742024-10-01T02:32:34.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 541.177846ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
12752024-10-01T02:32:34.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 1.007690522s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
12762024-10-01T02:32:34.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.408068998s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
12772024-10-01T02:32:34.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 689.269321ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
12782024-10-01T02:32:34.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.178086283s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
12792024-10-01T02:32:34.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 1.360913273s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
12802024-10-01T02:32:34.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 625.56553ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
12812024-10-01T02:32:34.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 1.286212637s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
12822024-10-01T02:32:34.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 814.830193ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
12832024-10-01T02:32:34.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 1.312428196s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
12842024-10-01T02:32:35.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.161961866s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
12852024-10-01T02:32:35.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 1.290471554s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
12862024-10-01T02:32:35.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 1.248953784s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
12872024-10-01T02:32:35.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 948.926906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
12882024-10-01T02:32:35.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 1.037125769s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
12892024-10-01T02:32:35.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.01130002s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
12902024-10-01T02:32:35.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 1.172850332s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
12912024-10-01T02:32:35.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 1.00518259s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
12922024-10-01T02:32:35.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 882.158298ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
12932024-10-01T02:32:35.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 1.415888551s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
12942024-10-01T02:32:35.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 1.346217611s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
12952024-10-01T02:32:35.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.345124301s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
12962024-10-01T02:32:35.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 638.199402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
12972024-10-01T02:32:35.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.173688068s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
12982024-10-01T02:32:35.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.134636492s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
12992024-10-01T02:32:36.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 551.14249ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
13002024-10-01T02:32:36.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 1.330005832s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
13012024-10-01T02:32:36.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.022605731s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
13022024-10-01T02:32:36.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 1.0536441s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
13032024-10-01T02:32:36.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 694.988554ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
13042024-10-01T02:32:36.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 965.931594ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
13052024-10-01T02:32:36.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 608.233215ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
13062024-10-01T02:32:36.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 943.630346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
13072024-10-01T02:32:36.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 952.871005ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
13082024-10-01T02:32:36.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 1.328928623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
13092024-10-01T02:32:36.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 1.423476578s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
13102024-10-01T02:32:36.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 810.002764ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
13112024-10-01T02:32:36.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 918.70346ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
13122024-10-01T02:32:36.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 570.513749ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
13132024-10-01T02:32:36.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 848.01545ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
13142024-10-01T02:32:37.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 865.135605ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
13152024-10-01T02:32:37.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.179582084s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
13162024-10-01T02:32:37.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 849.280003ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
13172024-10-01T02:32:37.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 1.465430696s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
13182024-10-01T02:32:37.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 1.074692083s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
13192024-10-01T02:32:37.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.344814502s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
13202024-10-01T02:32:37.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 692.046173ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
13212024-10-01T02:32:37.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.043883033s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
13222024-10-01T02:32:37.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 1.089564515s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
13232024-10-01T02:32:37.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 873.783506ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
13242024-10-01T02:32:37.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 1.466145845s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
13252024-10-01T02:32:37.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.264513859s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
13262024-10-01T02:32:37.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 610.774573ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
13272024-10-01T02:32:37.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 687.289281ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
13282024-10-01T02:32:37.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 971.120017ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
13292024-10-01T02:32:38.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.059157189s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
13302024-10-01T02:32:38.398ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.272243903s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
13312024-10-01T02:32:38.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 829.054132ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
13322024-10-01T02:32:38.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.267230695s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
13332024-10-01T02:32:38.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 644.412631ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
13342024-10-01T02:32:38.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 954.601631ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
13352024-10-01T02:32:38.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 1.211465497s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
13362024-10-01T02:32:38.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 985.874899ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
13372024-10-01T02:32:38.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 886.648094ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
13382024-10-01T02:32:38.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 1.425353178s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
13392024-10-01T02:32:38.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 862.937992ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
13402024-10-01T02:32:38.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 549.921617ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
13412024-10-01T02:32:38.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.084443209s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
13422024-10-01T02:32:38.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 598.701003ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
13432024-10-01T02:32:38.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 841.04316ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
13442024-10-01T02:32:38.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.432837181s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
13452024-10-01T02:32:38.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 737.640388ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
13462024-10-01T02:32:38.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 996.641846ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
13472024-10-01T02:32:39.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 908.047884ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
13482024-10-01T02:32:39.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 767.821361ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
13492024-10-01T02:32:39.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 997.100562ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
13502024-10-01T02:32:39.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 1.041628623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
13512024-10-01T02:32:39.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 852.936438ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
13522024-10-01T02:32:39.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.499908855s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
13532024-10-01T02:32:39.457ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 1.365765249s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
13542024-10-01T02:32:39.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 713.180377ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
13552024-10-01T02:32:39.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 1.074860095s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
13562024-10-01T02:32:39.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 692.735981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
13572024-10-01T02:32:39.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 749.543877ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
13582024-10-01T02:32:39.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.314997502s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
13592024-10-01T02:32:39.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 555.372677ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
13602024-10-01T02:32:39.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 960.152076ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
13612024-10-01T02:32:39.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.030117902s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
13622024-10-01T02:32:39.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 582.127919ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
13632024-10-01T02:32:39.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 1.404098883s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
13642024-10-01T02:32:40.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 925.489668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
13652024-10-01T02:32:40.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 848.129457ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
13662024-10-01T02:32:40.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 1.142441178s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
13672024-10-01T02:32:40.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 812.452895ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
13682024-10-01T02:32:40.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 903.453622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
13692024-10-01T02:32:40.219ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13702024-10-01T02:32:40.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 766.967432ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
13712024-10-01T02:32:40.241ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
13722024-10-01T02:32:40.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 1.265294854s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
13732024-10-01T02:32:40.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 774.323023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
13742024-10-01T02:32:40.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 1.471599579s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
13752024-10-01T02:32:40.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 1.114200604s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
13762024-10-01T02:32:40.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 560.357731ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
13772024-10-01T02:32:40.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 60.343964ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
13782024-10-01T02:32:40.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 70.329081ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
13792024-10-01T02:32:40.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 521.655606ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
13802024-10-01T02:32:40.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 132.656008ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
13812024-10-01T02:32:40.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 798.016788ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
13822024-10-01T02:32:40.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 547.912369ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
13832024-10-01T02:32:40.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 371.959029ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
13842024-10-01T02:32:40.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 546.528295ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
13852024-10-01T02:32:40.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.249020899s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
13862024-10-01T02:32:40.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 907.917009ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
13872024-10-01T02:32:40.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 645.439403ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
13882024-10-01T02:32:40.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 538.836542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
13892024-10-01T02:32:41.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.350771611s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
13902024-10-01T02:32:41.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 1.333162585s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
13912024-10-01T02:32:41.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 1.142831062s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
13922024-10-01T02:32:41.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.437138018s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
13932024-10-01T02:32:41.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 882.944191ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
13942024-10-01T02:32:41.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 745.525351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
13952024-10-01T02:32:41.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 405.459886ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
13962024-10-01T02:32:41.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 680.416369ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
13972024-10-01T02:32:41.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 535.278794ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
13982024-10-01T02:32:41.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 733.45337ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
13992024-10-01T02:32:41.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 602.173552ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
14002024-10-01T02:32:41.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 1.021444375s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
14012024-10-01T02:32:41.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 1.232287558s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
14022024-10-01T02:32:41.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 1.363703862s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
14032024-10-01T02:32:41.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 1.250932032s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
14042024-10-01T02:32:41.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 830.726546ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
14052024-10-01T02:32:41.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 1.015216856s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14062024-10-01T02:32:41.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 803.816518ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
14072024-10-01T02:32:41.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 610.986705ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
14082024-10-01T02:32:41.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 1.132815064s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
14092024-10-01T02:32:41.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 552.758739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
14102024-10-01T02:32:41.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 1.050250736s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
14112024-10-01T02:32:42.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 1.008088754s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
14122024-10-01T02:32:42.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.285551044s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
14132024-10-01T02:32:42.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 996.857342ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
14142024-10-01T02:32:42.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 989.917269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
14152024-10-01T02:32:42.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 721.84961ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
14162024-10-01T02:32:42.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 1.28840822s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
14172024-10-01T02:32:42.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.146934688s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
14182024-10-01T02:32:42.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 738.016102ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
14192024-10-01T02:32:42.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 981.319467ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
14202024-10-01T02:32:42.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 947.895907ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
14212024-10-01T02:32:42.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 1.036522112s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
14222024-10-01T02:32:42.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 521.090736ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
14232024-10-01T02:32:42.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049") failed: Property not found. retry in 1.497215707s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
14242024-10-01T02:32:42.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 1.40405306s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14252024-10-01T02:32:42.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af") failed: Property not found. retry in 1.411080336s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
14262024-10-01T02:32:42.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17") failed: Property not found. retry in 1.455553126s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
14272024-10-01T02:32:42.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 590.610631ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
14282024-10-01T02:32:42.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 916.104038ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
14292024-10-01T02:32:43.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697") failed: Property not found. retry in 935.858297ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
14302024-10-01T02:32:43.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d") failed: Property not found. retry in 1.402618828s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
14312024-10-01T02:32:43.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 884.092055ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
14322024-10-01T02:32:43.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee") failed: Property not found. retry in 1.15485544s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
14332024-10-01T02:32:43.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63") failed: Property not found. retry in 1.45728342s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
14342024-10-01T02:32:43.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252") failed: Property not found. retry in 1.278620333s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
14352024-10-01T02:32:43.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a") failed: Property not found. retry in 1.472452214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
14362024-10-01T02:32:43.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6") failed: Property not found. retry in 1.132080525s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
14372024-10-01T02:32:43.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290") failed: Property not found. retry in 1.136980551s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
14382024-10-01T02:32:43.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1") failed: Property not found. retry in 1.127292518s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
14392024-10-01T02:32:43.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35") failed: Property not found. retry in 1.353159007s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
14402024-10-01T02:32:43.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644") failed: Property not found. retry in 1.464883916s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
14412024-10-01T02:32:43.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a") failed: Property not found. retry in 771.191991ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
14422024-10-01T02:32:43.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9") failed: Property not found. retry in 960.723718ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
14432024-10-01T02:32:43.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015") failed: Property not found. retry in 1.099420115s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
14442024-10-01T02:32:43.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807") failed: Property not found. retry in 1.180208104s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
14452024-10-01T02:32:44.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 656.219307ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14462024-10-01T02:32:44.712ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 8.034863017s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
14472024-10-01T02:32:44.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 580.055428ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14482024-10-01T02:32:45.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 752.403135ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14492024-10-01T02:32:45.927ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57885
14502024-10-01T02:32:45.932ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 540 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57885 req_id = 966cc1de-1815-484b-81da-fd17f855bcc8 response_code = 200 uri = /baseboard
14512024-10-01T02:32:46.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 853.168837ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14522024-10-01T02:32:47.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 1.197170516s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14532024-10-01T02:32:48.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 1.402178345s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14542024-10-01T02:32:49.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f") failed: Property not found. retry in 1.478855235s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14552024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_clickhouse_9df68fe6-52e1-420d-8e69-ae187fac2b5f
14562024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_external_dns_eb937866-c779-4a9e-aec4-cc18302d4ce1
14572024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_oximeter_41f2943e-6041-44c9-a47a-3cf5fb012644
14582024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_46e1cda7-578a-4867-bb73-b6c56c6231af
14592024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_9cf5c3af-b82a-4f8e-8aba-3e39ebd2b64a
14602024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_772acfa5-ee20-4a17-a053-c207ffa50ed6
14612024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_44dd83be-d555-4f7e-91d1-342fa869f697
14622024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_3ae874d1-bf62-481b-aa5f-911ac7586252
14632024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_5ee3adc6-bb70-4180-8a0e-1e58aba9f807
14642024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_cdeda3ef-54c4-466c-bcd2-15530c779f17
14652024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_3dfd7033-71d8-4eb9-bb97-49dae3b5cd35
14662024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_5ab93dca-3349-4e95-8adc-97fe1d9e5049
14672024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_0abff335-deb2-4991-b5b4-a3ef2f565fa9
14682024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_c516b690-7599-4aaa-8432-aa06ded5383a
14692024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_b4924e44-5dc5-45e0-a39b-c430868b3e63
14702024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_7f46a1e4-9d58-499a-94fb-494a9375a290
14712024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_99df6194-3c68-48e6-a934-d49d19c5c7ee
14722024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_external_dns_0a1ed896-5078-4267-b636-10151376f015
14732024-10-01T02:32:51.138ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_e9e4c757-e076-4386-92e2-8217b8bb307d
14742024-10-01T02:32:51.140ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
14752024-10-01T02:32:51.142ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
14762024-10-01T02:32:51.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 35265546 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:44791 req_id = 70cc5a02-a985-402e-9b7c-ee5723cd71b1 response_code = 204 uri = /omicron-zones
14772024-10-01T02:32:51.149ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1322
14782024-10-01T02:32:51.150ZINFOSledAgent (RSS): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
14792024-10-01T02:32:51.150ZINFOSledAgent (RSS): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
14802024-10-01T02:32:51.152ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
14812024-10-01T02:32:51.153ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:720
14822024-10-01T02:32:51.186ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:887
14832024-10-01T02:32:51.196ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14842024-10-01T02:32:51.323ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14852024-10-01T02:32:51.495ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14862024-10-01T02:32:51.698ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14872024-10-01T02:32:51.906ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14882024-10-01T02:32:52.158ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14892024-10-01T02:32:52.319ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14902024-10-01T02:32:52.516ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14912024-10-01T02:32:52.734ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14922024-10-01T02:32:52.781ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.197154526s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
14932024-10-01T02:32:52.782ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
14942024-10-01T02:32:53.014ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
14952024-10-01T02:32:53.122ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14962024-10-01T02:32:53.507ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
14972024-10-01T02:32:53.898ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
14982024-10-01T02:32:54.018ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
14992024-10-01T02:32:54.257ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15002024-10-01T02:32:54.871ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15012024-10-01T02:32:55.021ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15022024-10-01T02:32:55.503ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15032024-10-01T02:32:55.816ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15042024-10-01T02:32:56.024ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15052024-10-01T02:32:57.027ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15062024-10-01T02:32:57.328ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15072024-10-01T02:32:57.504ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15082024-10-01T02:32:58.025ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15092024-10-01T02:32:58.031ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15102024-10-01T02:32:59.033ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15112024-10-01T02:32:59.503ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15122024-10-01T02:32:59.990ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15132024-10-01T02:33:00.036ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15142024-10-01T02:33:01.040ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15152024-10-01T02:33:01.115ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15162024-10-01T02:33:01.503ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15172024-10-01T02:33:02.012ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 20.215887841s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15182024-10-01T02:33:02.043ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15192024-10-01T02:33:02.285ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15202024-10-01T02:33:03.046ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15212024-10-01T02:33:03.504ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15222024-10-01T02:33:04.049ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15232024-10-01T02:33:05.052ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15242024-10-01T02:33:05.503ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15252024-10-01T02:33:05.890ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/52547763-7759-42e1-81a9-8b833f59f5dd/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
15262024-10-01T02:33:06.056ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15272024-10-01T02:33:07.058ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/probe_manager.rs:230 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15282024-10-01T02:33:07.504ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/2a571ec0-85a3-450e-88fc-680c13ffa211) file = sled-agent/src/nexus.rs:571 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15292024-10-01T02:33:08.624ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 403 Forbidden; headers: {"content-type": "application/json", "x-request-id": "005a1c70-0dc7-4be4-b02c-c8b32d0a2a11", "content-length": "113", "date": "Tue, 01 Oct 2024 02:33:08 GMT"}; value: Error { error_code: Some("Forbidden"), message: "Forbidden", request_id: "005a1c70-0dc7-4be4-b02c-c8b32d0a2a11" } file = sled-agent/src/rack_setup/service.rs:949
15302024-10-01T02:33:11.568ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
15312024-10-01T02:33:11.722ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45936
15322024-10-01T02:33:11.722ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47370
15332024-10-01T02:33:11.723ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 412 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47370 req_id = a2bbd35f-0e2c-4c08-9617-af8c0babbcd0 response_code = 200 uri = /v2p
15342024-10-01T02:33:11.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3308 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45936 req_id = 489e4538-d1bd-4a23-8242-8ba219063c99 response_code = 200 uri = /vpc-routes
15352024-10-01T02:33:11.829ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7412 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45936 req_id = 03b16f70-2f46-4389-b9e8-e16364c702fa response_code = 204 uri = /vpc-routes
15362024-10-01T02:33:11.858ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33765
15372024-10-01T02:33:11.858ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33765 req_id = e37cc2a5-5772-4218-8ca6-85ae1fda9137 response_code = 200 uri = /v2p
15382024-10-01T02:33:12.156ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40887
15392024-10-01T02:33:12.156ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15402024-10-01T02:33:12.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 209467 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40887 req_id = 54ccc9db-0f68-498e-aa2d-26d00876f990 response_code = 200 uri = /inventory
15412024-10-01T02:33:12.365ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65329
15422024-10-01T02:33:12.365ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15432024-10-01T02:33:12.370ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15442024-10-01T02:33:12.370ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-zones.json file = common/src/ledger.rs:177
15452024-10-01T02:33:12.372ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-zones.json file = common/src/ledger.rs:177
15462024-10-01T02:33:12.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3968 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40887 req_id = 4f43e7c3-5b75-450f-a546-c81192bb949e response_code = 200 uri = /omicron-zones
15472024-10-01T02:33:12.584ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 218598 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65329 req_id = 43c73a58-03cd-4374-8fdb-1523775f03eb response_code = 200 uri = /inventory
15482024-10-01T02:33:12.584ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15492024-10-01T02:33:12.584ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-zones.json file = common/src/ledger.rs:177
15502024-10-01T02:33:12.585ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-zones.json file = common/src/ledger.rs:177
15512024-10-01T02:33:12.586ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2097 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65329 req_id = 90f28937-fedc-4ce7-8e7d-a5cdf2a43010 response_code = 200 uri = /omicron-zones
15522024-10-01T02:33:14.235ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52470
15532024-10-01T02:33:14.236ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15542024-10-01T02:33:14.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
15552024-10-01T02:33:14.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15562024-10-01T02:33:14.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15572024-10-01T02:33:14.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15582024-10-01T02:33:14.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15592024-10-01T02:33:14.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15602024-10-01T02:33:14.237ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2045 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52470 req_id = 3e28ccb1-7723-449a-b010-76dfb72b73ea response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15612024-10-01T02:33:14.239ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:959
15622024-10-01T02:33:14.614ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49823
15632024-10-01T02:33:14.615ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15642024-10-01T02:33:14.615ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
15652024-10-01T02:33:14.615ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15662024-10-01T02:33:14.615ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15672024-10-01T02:33:14.615ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15682024-10-01T02:33:14.615ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15692024-10-01T02:33:14.615ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15702024-10-01T02:33:14.615ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 365 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49823 req_id = ace5f580-cd1e-4483-acbc-acc87a11e9e3 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15712024-10-01T02:33:14.646ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33688
15722024-10-01T02:33:14.647ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15732024-10-01T02:33:14.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
15742024-10-01T02:33:14.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15752024-10-01T02:33:14.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15762024-10-01T02:33:14.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15772024-10-01T02:33:14.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15782024-10-01T02:33:14.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15792024-10-01T02:33:14.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 834 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33688 req_id = 735a828d-668c-4cf4-9379-d0a886e0b6af response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15802024-10-01T02:33:14.665ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41125
15812024-10-01T02:33:14.666ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15822024-10-01T02:33:14.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
15832024-10-01T02:33:14.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15842024-10-01T02:33:14.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15852024-10-01T02:33:14.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15862024-10-01T02:33:14.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15872024-10-01T02:33:14.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15882024-10-01T02:33:14.666ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 248 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41125 req_id = b0010267-0513-4e73-8428-ef74542acb46 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15892024-10-01T02:33:14.707ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64892
15902024-10-01T02:33:14.707ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15912024-10-01T02:33:14.908ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 200973 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64892 req_id = 53fd661f-4698-48a5-b676-27649b1847f6 response_code = 200 uri = /inventory
15922024-10-01T02:33:14.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15932024-10-01T02:33:14.908ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/config/omicron-zones.json file = common/src/ledger.rs:177
15942024-10-01T02:33:14.909ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/config/omicron-zones.json file = common/src/ledger.rs:177
15952024-10-01T02:33:14.909ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 770 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64892 req_id = a961f4f2-cfcd-4712-99df-d98fc1762836 response_code = 200 uri = /omicron-zones
15962024-10-01T02:33:15.988ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35319
15972024-10-01T02:33:15.988ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35319 req_id = 4cb36c7a-ba2a-476c-827f-eb252fc9f204 response_code = 200 uri = /baseboard
15982024-10-01T02:33:20.404ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64805
15992024-10-01T02:33:20.404ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64805 req_id = dfdf73be-5411-42ba-932f-79c3f37d61e9 response_code = 200 uri = /vpc-routes
16002024-10-01T02:33:20.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64805 req_id = 7d6ef6cb-9eff-495f-8798-77d258fe10c0 response_code = 204 uri = /vpc-routes
16012024-10-01T02:33:41.297ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38326
16022024-10-01T02:33:41.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38326 req_id = 5dcaede6-6584-4916-b033-2d72503a7bc4 response_code = 200 uri = /vpc-routes
16032024-10-01T02:33:41.338ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38326 req_id = 1081b86d-c140-4ddf-a68f-8184b045264f response_code = 204 uri = /vpc-routes
16042024-10-01T02:33:41.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48030
16052024-10-01T02:33:41.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48030 req_id = 946bdef4-ae33-4bed-8a3b-03344c88ffce response_code = 200 uri = /vpc-routes
16062024-10-01T02:33:41.421ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58707
16072024-10-01T02:33:41.421ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58707 req_id = 831a834c-b1dd-45bd-877c-7f6cd5fff175 response_code = 200 uri = /v2p
16082024-10-01T02:33:41.435ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 749 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48030 req_id = 00410947-05a4-42c0-a0fd-a56f5942ba88 response_code = 204 uri = /vpc-routes
16092024-10-01T02:33:41.503ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38744
16102024-10-01T02:33:41.504ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 877 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38744 req_id = 70215e8f-8eb2-455e-87b6-7bd6e7ad910b response_code = 200 uri = /v2p
16112024-10-01T02:33:41.641ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65059
16122024-10-01T02:33:41.650ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16132024-10-01T02:33:41.908ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16142024-10-01T02:33:42.020ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 379465 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65059 req_id = bf7e84fb-86c2-44f4-b6ca-c6f14abf866f response_code = 204 uri = /switch-ports
16152024-10-01T02:33:42.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57497
16162024-10-01T02:33:42.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58584
16172024-10-01T02:33:42.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53536
16182024-10-01T02:33:42.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58595
16192024-10-01T02:33:42.021ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16202024-10-01T02:33:42.021ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 744 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57497 req_id = e907d4f2-4a4c-419b-a194-35066cf4b18c response_code = 200 uri = /vpc-routes
16212024-10-01T02:33:42.025ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3731 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53536 req_id = c5fb6364-1ade-4143-bc21-13424ed7cf6a response_code = 200 uri = /v2p
16222024-10-01T02:33:42.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 10209 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65059 req_id = 81c823dc-14a9-49bd-a856-09f41db66aec response_code = 200 uri = /network-bootstore-config
16232024-10-01T02:33:42.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57497 req_id = 7174f880-1e71-4def-b412-14e207002a51 response_code = 204 uri = /vpc-routes
16242024-10-01T02:33:42.068ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16252024-10-01T02:33:42.149ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16262024-10-01T02:33:42.149ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 128870 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58595 req_id = 5b8f3348-b83a-4875-93c2-ed1674a7f0f4 response_code = 204 uri = /switch-ports
16272024-10-01T02:33:42.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 491 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58595 req_id = d74a4216-4cb3-44e6-9721-314a574f9a32 response_code = 200 uri = /network-bootstore-config
16282024-10-01T02:33:42.213ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16292024-10-01T02:33:42.252ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
16302024-10-01T02:33:42.256ZINFOSledAgent (bootstore): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
16312024-10-01T02:33:42.269ZINFOSledAgent (bootstore): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
16322024-10-01T02:33:42.270ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
16332024-10-01T02:33:42.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 27919 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65059 req_id = 7601b124-6d15-4087-a24f-2b107f7a2670 response_code = 204 uri = /network-bootstore-config
16342024-10-01T02:33:42.277ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 256066 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58584 req_id = 6a0f012e-72c1-4cc3-ae97-efd791eec186 response_code = 204 uri = /switch-ports
16352024-10-01T02:33:42.277ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 153 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58584 req_id = 65268535-24a5-4136-8e14-87d1db590c84 response_code = 200 uri = /network-bootstore-config
16362024-10-01T02:33:42.375ZINFOSledAgent (bootstore): Attempting to update network config with generation: 3, current_generation: 2 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
16372024-10-01T02:33:42.375ZINFOSledAgent (bootstore): Writing ledger to /pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
16382024-10-01T02:33:42.375ZINFOSledAgent (bootstore): Writing ledger to /pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
16392024-10-01T02:33:42.375ZINFOSledAgent (bootstore): Broadcasting network config with generation 3 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
16402024-10-01T02:33:42.375ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 607 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58595 req_id = cefe2b50-7284-44d5-ba10-981f56d47f48 response_code = 204 uri = /network-bootstore-config
16412024-10-01T02:33:46.021ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62934
16422024-10-01T02:33:46.021ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62934 req_id = a2fd8ddf-c151-41bf-a809-038e0d6aa754 response_code = 200 uri = /baseboard
16432024-10-01T02:33:52.358ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:45292 remote_addr = [fd00:1122:3344:101::d]:60055 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211
16442024-10-01T02:33:52.392ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 33294 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 32ef6030-cfae-44df-8a80-4704274f21fa response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
16452024-10-01T02:34:11.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57970
16462024-10-01T02:34:11.258ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57970 req_id = c12394b8-f54e-435f-90fb-088be3a572a1 response_code = 200 uri = /vpc-routes
16472024-10-01T02:34:11.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62647
16482024-10-01T02:34:11.262ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62647 req_id = 366c69d3-18ab-4155-9724-044d7dca86dd response_code = 200 uri = /vpc-routes
16492024-10-01T02:34:11.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57970 req_id = 9b6f572d-e161-4342-87f8-d76067d16098 response_code = 204 uri = /vpc-routes
16502024-10-01T02:34:11.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62647 req_id = b01380a0-8423-4a9e-9490-565815e89702 response_code = 204 uri = /vpc-routes
16512024-10-01T02:34:11.409ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42951
16522024-10-01T02:34:11.410ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16532024-10-01T02:34:11.451ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16542024-10-01T02:34:11.513ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103522 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42951 req_id = 916ca256-e652-4d17-9863-3d682cf489b4 response_code = 204 uri = /switch-ports
16552024-10-01T02:34:11.513ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34255
16562024-10-01T02:34:11.513ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50481
16572024-10-01T02:34:11.514ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 403 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42951 req_id = 4004c493-6d0f-42eb-9e7d-f55da13358bb response_code = 200 uri = /network-bootstore-config
16582024-10-01T02:34:11.514ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 456 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50481 req_id = 204113c4-8972-4f2b-845a-6fc84cc424fe response_code = 200 uri = /v2p
16592024-10-01T02:34:11.514ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 362 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34255 req_id = ffc090ad-0218-466e-acc3-a412cf6eb110 response_code = 200 uri = /v2p
16602024-10-01T02:34:11.698ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58021
16612024-10-01T02:34:11.699ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16622024-10-01T02:34:11.740ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16632024-10-01T02:34:11.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110188 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58021 req_id = 6eb9012c-2de8-48de-8741-aebef5632335 response_code = 204 uri = /switch-ports
16642024-10-01T02:34:11.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58021 req_id = 4c29a42a-2898-489d-a64c-1a2633871fac response_code = 200 uri = /network-bootstore-config
16652024-10-01T02:34:11.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33162
16662024-10-01T02:34:11.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33162 req_id = daf35440-4db1-4940-8222-b0141c2017e7 response_code = 200 uri = /vpc-routes
16672024-10-01T02:34:11.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33162 req_id = 0e049b3b-85ed-447d-8fc2-e6647ec29ad2 response_code = 204 uri = /vpc-routes
16682024-10-01T02:34:12.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42742
16692024-10-01T02:34:12.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42742 req_id = 0da334e1-9e4d-4f3b-b80b-6dd8bb77c0f7 response_code = 200 uri = /v2p
16702024-10-01T02:34:12.149ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40487
16712024-10-01T02:34:12.149ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16722024-10-01T02:34:12.190ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16732024-10-01T02:34:12.247ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98028 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40487 req_id = 1816a497-703c-4658-b1ae-319d5145d7af response_code = 204 uri = /switch-ports
16742024-10-01T02:34:12.247ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40487 req_id = 445f56da-679a-4474-b1ce-8c3ea9dbbbfe response_code = 200 uri = /network-bootstore-config
16752024-10-01T02:34:16.054ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52694
16762024-10-01T02:34:16.054ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52694 req_id = 6d59167e-e235-4333-939d-731bfcb99330 response_code = 200 uri = /baseboard
16772024-10-01T02:34:22.366ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7584 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = d30cc515-185f-47ee-9cbc-119d3a574583 response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
16782024-10-01T02:34:41.254ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54740
16792024-10-01T02:34:41.254ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54740 req_id = c4bae2f5-3879-4817-a41a-3a32559237ea response_code = 200 uri = /vpc-routes
16802024-10-01T02:34:41.267ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1020 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54740 req_id = aba1a595-e3f1-4840-ba4b-c28ff07d7762 response_code = 204 uri = /vpc-routes
16812024-10-01T02:34:41.267ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64778
16822024-10-01T02:34:41.267ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64778 req_id = d5c996f4-987e-4cfa-ba46-ab036d99fb1f response_code = 200 uri = /vpc-routes
16832024-10-01T02:34:41.276ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64778 req_id = bce761bd-de58-4ab3-b0a4-585e4ab9a423 response_code = 204 uri = /vpc-routes
16842024-10-01T02:34:41.343ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49249
16852024-10-01T02:34:41.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 649 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49249 req_id = 4c95d93b-3640-49ed-8243-931170a9c07b response_code = 200 uri = /v2p
16862024-10-01T02:34:41.360ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37672
16872024-10-01T02:34:41.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37672 req_id = a7d72344-bcb0-4224-bd74-ca05c6959108 response_code = 200 uri = /v2p
16882024-10-01T02:34:41.393ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52897
16892024-10-01T02:34:41.393ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16902024-10-01T02:34:41.430ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16912024-10-01T02:34:41.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93723 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52897 req_id = 425f20ff-a8af-4904-bc0d-9caa248e784a response_code = 204 uri = /switch-ports
16922024-10-01T02:34:41.487ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63427
16932024-10-01T02:34:41.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 215 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52897 req_id = 3bd32ede-6c3a-45e2-b257-813e1ba4b848 response_code = 200 uri = /network-bootstore-config
16942024-10-01T02:34:41.487ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
16952024-10-01T02:34:41.539ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
16962024-10-01T02:34:41.615ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 128047 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63427 req_id = 848fd110-cb01-471f-9f69-37e7723055bf response_code = 204 uri = /switch-ports
16972024-10-01T02:34:41.616ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 630 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63427 req_id = d74b5ac4-00df-4bcb-b260-64f97d1506d1 response_code = 200 uri = /network-bootstore-config
16982024-10-01T02:34:41.722ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55672
16992024-10-01T02:34:41.722ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55672 req_id = d973ec47-7861-41e3-bcce-d83950028c99 response_code = 200 uri = /vpc-routes
17002024-10-01T02:34:41.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55672 req_id = 3a70600c-c5cd-484d-8dd3-9ec3889d2efd response_code = 204 uri = /vpc-routes
17012024-10-01T02:34:41.849ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38379
17022024-10-01T02:34:41.849ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38379 req_id = 815e083d-2d50-4c45-9347-b4fcaaadefe2 response_code = 200 uri = /v2p
17032024-10-01T02:34:41.862ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58627
17042024-10-01T02:34:41.862ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17052024-10-01T02:34:41.897ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17062024-10-01T02:34:41.950ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88235 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58627 req_id = 93147ea8-675b-43b7-9c8b-94cacfca6953 response_code = 204 uri = /switch-ports
17072024-10-01T02:34:41.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58627 req_id = 48db6a09-ae54-46ec-bae8-67fe9f13ed5d response_code = 200 uri = /network-bootstore-config
17082024-10-01T02:34:46.089ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62430
17092024-10-01T02:34:46.089ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 67 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62430 req_id = 7fe4d865-270c-4b07-8363-cbf867663e8a response_code = 200 uri = /baseboard
17102024-10-01T02:34:52.367ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8366 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = e371a882-94b1-44ba-a2be-56f0fdd8b819 response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
17112024-10-01T02:35:11.363ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42708
17122024-10-01T02:35:11.363ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42708 req_id = 655f536e-5b59-4cf4-b2cb-6ea3704fb7d9 response_code = 200 uri = /vpc-routes
17132024-10-01T02:35:11.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:42708 req_id = 468127c5-9178-4793-9610-7829c433bc15 response_code = 204 uri = /vpc-routes
17142024-10-01T02:35:11.448ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41548
17152024-10-01T02:35:11.449ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41548 req_id = f53785f2-49ff-4d77-b631-f9604d71bd22 response_code = 200 uri = /v2p
17162024-10-01T02:35:11.515ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54399
17172024-10-01T02:35:11.515ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54399 req_id = b6d6b0ea-b326-4b13-9fdd-b5062b1adaf2 response_code = 200 uri = /v2p
17182024-10-01T02:35:11.603ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56457
17192024-10-01T02:35:11.603ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56457 req_id = 9654852a-2158-46ec-8627-903633b45748 response_code = 200 uri = /vpc-routes
17202024-10-01T02:35:11.622ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 753 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56457 req_id = 3e4096e1-30b3-4b12-a3ee-b1e918df596b response_code = 204 uri = /vpc-routes
17212024-10-01T02:35:11.630ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62153
17222024-10-01T02:35:11.631ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17232024-10-01T02:35:11.679ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17242024-10-01T02:35:11.743ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 112879 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62153 req_id = 29c18a6f-bf87-44cb-b893-6dd609fc1cc3 response_code = 204 uri = /switch-ports
17252024-10-01T02:35:11.744ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52447
17262024-10-01T02:35:11.744ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17272024-10-01T02:35:11.782ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17282024-10-01T02:35:11.845ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 101858 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52447 req_id = d734c1cd-771f-4cb3-addd-99fa2441b416 response_code = 204 uri = /switch-ports
17292024-10-01T02:35:11.846ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35985
17302024-10-01T02:35:11.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35985 req_id = 5596db06-ccb2-4a48-a1b5-48e8bc6fd1bd response_code = 200 uri = /vpc-routes
17312024-10-01T02:35:11.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 226 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62153 req_id = 813614b2-c00c-4b11-9b5f-7ce589b996a8 response_code = 200 uri = /network-bootstore-config
17322024-10-01T02:35:11.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52447 req_id = a013813c-6e74-4c66-bb47-34ac0f54682e response_code = 200 uri = /network-bootstore-config
17332024-10-01T02:35:11.855ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 153 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35985 req_id = bdf0a803-ad65-4c65-91a5-e849b830ed30 response_code = 204 uri = /vpc-routes
17342024-10-01T02:35:11.942ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45392
17352024-10-01T02:35:11.942ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45392 req_id = 89235afc-f0a8-44e9-8f6e-ae4138438eb9 response_code = 200 uri = /v2p
17362024-10-01T02:35:11.947ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57829
17372024-10-01T02:35:11.947ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17382024-10-01T02:35:11.984ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17392024-10-01T02:35:12.040ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93123 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57829 req_id = abc65ab6-4003-4b98-b4b1-60d8b4361733 response_code = 204 uri = /switch-ports
17402024-10-01T02:35:12.040ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57829 req_id = a2983cd3-78a5-4fdf-a84f-94520ac7cb2e response_code = 200 uri = /network-bootstore-config
17412024-10-01T02:35:16.125ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64963
17422024-10-01T02:35:16.125ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64963 req_id = b906e59b-2594-4b4f-8441-5517ed3e026e response_code = 200 uri = /baseboard
17432024-10-01T02:35:22.367ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7378 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = f4790a5f-156a-4837-bc32-56d217cf845e response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
17442024-10-01T02:35:26.281ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
17452024-10-01T02:35:41.275ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57349
17462024-10-01T02:35:41.275ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57349 req_id = 6350cfb2-1e87-47d4-9211-ba3fc61fff8b response_code = 200 uri = /vpc-routes
17472024-10-01T02:35:41.303ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63605
17482024-10-01T02:35:41.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63605 req_id = e7b15730-e7ee-4a04-aac4-470b6b4bc39a response_code = 200 uri = /vpc-routes
17492024-10-01T02:35:41.306ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41209
17502024-10-01T02:35:41.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 899 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41209 req_id = bb99b090-3f30-4404-a8c7-ad2efe8f16c7 response_code = 200 uri = /v2p
17512024-10-01T02:35:41.310ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57349 req_id = 20997e67-f94d-4b9f-bae7-27165bf1f479 response_code = 204 uri = /vpc-routes
17522024-10-01T02:35:41.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63605 req_id = 22acbdbb-677f-4637-aa8e-436a3b66a4a9 response_code = 204 uri = /vpc-routes
17532024-10-01T02:35:41.324ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35228
17542024-10-01T02:35:41.324ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17552024-10-01T02:35:41.365ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17562024-10-01T02:35:41.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 104120 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35228 req_id = d82a3eb6-369e-4af3-8aed-efc731b00245 response_code = 204 uri = /switch-ports
17572024-10-01T02:35:41.428ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52786
17582024-10-01T02:35:41.428ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35077
17592024-10-01T02:35:41.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 264 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35228 req_id = 7f948a0e-527e-47b4-a26d-84a729145349 response_code = 200 uri = /network-bootstore-config
17602024-10-01T02:35:41.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 246 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52786 req_id = dc9b399b-6677-409c-9615-86c0de5826ca response_code = 200 uri = /v2p
17612024-10-01T02:35:41.428ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17622024-10-01T02:35:41.468ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17632024-10-01T02:35:41.542ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 113756 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35077 req_id = 1253fa35-16b2-4a45-b977-6f633dc94961 response_code = 204 uri = /switch-ports
17642024-10-01T02:35:41.542ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35077 req_id = 75fdf42c-9260-49a5-93ea-035cddfa6533 response_code = 200 uri = /network-bootstore-config
17652024-10-01T02:35:41.676ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51115
17662024-10-01T02:35:41.676ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51115 req_id = 047e89c0-cadd-4c2b-a903-8d3031762b83 response_code = 200 uri = /vpc-routes
17672024-10-01T02:35:41.685ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51115 req_id = a1c5974a-355f-4280-a875-6d864a53673f response_code = 204 uri = /vpc-routes
17682024-10-01T02:35:41.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44050
17692024-10-01T02:35:41.767ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17702024-10-01T02:35:41.807ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17712024-10-01T02:35:41.867ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100221 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44050 req_id = b20083a7-f964-432a-b384-71a31923e701 response_code = 204 uri = /switch-ports
17722024-10-01T02:35:41.867ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55160
17732024-10-01T02:35:41.868ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 967 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44050 req_id = 8ecc6b5f-6351-483e-9263-b921bf9aa0ed response_code = 200 uri = /network-bootstore-config
17742024-10-01T02:35:41.868ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 870 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55160 req_id = ccd72c59-8feb-47f2-b61f-610161f94e08 response_code = 200 uri = /v2p
17752024-10-01T02:35:46.158ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54280
17762024-10-01T02:35:46.158ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 644 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54280 req_id = a85921b3-df19-415f-9816-e2f8b85a1138 response_code = 200 uri = /baseboard
17772024-10-01T02:35:52.371ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7726 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = cee51c88-f0fc-43a6-9bd9-4a462bc97ce0 response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
17782024-10-01T02:36:11.367ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56401
17792024-10-01T02:36:11.368ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56401 req_id = 9beafba6-5fc3-47fd-9fc5-e33e290a5859 response_code = 200 uri = /vpc-routes
17802024-10-01T02:36:11.377ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56401 req_id = 13d07aa4-f58f-44ca-9a09-654fbed601d6 response_code = 204 uri = /vpc-routes
17812024-10-01T02:36:11.414ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41367
17822024-10-01T02:36:11.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 27 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41367 req_id = 0f9cf875-0c29-440e-99d5-61e0873c1092 response_code = 200 uri = /vpc-routes
17832024-10-01T02:36:11.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41367 req_id = 2a70ea1e-e6aa-470d-9de3-d4e6c5d1e890 response_code = 204 uri = /vpc-routes
17842024-10-01T02:36:11.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37613
17852024-10-01T02:36:11.537ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 238 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37613 req_id = 953120e7-465d-4119-887a-6b587b925897 response_code = 200 uri = /v2p
17862024-10-01T02:36:11.576ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59319
17872024-10-01T02:36:11.576ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17882024-10-01T02:36:11.628ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17892024-10-01T02:36:11.694ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 117873 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59319 req_id = a416949b-31d1-4039-9258-9b37ce6c510d response_code = 204 uri = /switch-ports
17902024-10-01T02:36:11.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44471
17912024-10-01T02:36:11.695ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59885
17922024-10-01T02:36:11.695ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 200 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59319 req_id = a95e16fb-89b8-4ec1-89ed-b4f7c3399d9c response_code = 200 uri = /network-bootstore-config
17932024-10-01T02:36:11.695ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
17942024-10-01T02:36:11.699ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4170 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44471 req_id = f308a7ce-93ed-4717-ac1e-e3548c1619ba response_code = 200 uri = /v2p
17952024-10-01T02:36:11.735ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
17962024-10-01T02:36:11.786ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34921
17972024-10-01T02:36:11.786ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34921 req_id = 430dd041-70cd-466a-bf8f-0eb885a83f96 response_code = 200 uri = /vpc-routes
17982024-10-01T02:36:11.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100510 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59885 req_id = f626f4e6-ba5f-46fe-a782-44479f114f94 response_code = 204 uri = /switch-ports
17992024-10-01T02:36:11.796ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 997 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59885 req_id = fa184cf4-50ba-409c-86c4-86f872d4406d response_code = 200 uri = /network-bootstore-config
18002024-10-01T02:36:11.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34921 req_id = ff43c670-b66f-4fea-a628-fc3906555722 response_code = 204 uri = /vpc-routes
18012024-10-01T02:36:11.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50145
18022024-10-01T02:36:11.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50145 req_id = 351cf6e3-a37a-4d29-8a10-969e3c59db1e response_code = 200 uri = /v2p
18032024-10-01T02:36:12.077ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46506
18042024-10-01T02:36:12.078ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18052024-10-01T02:36:12.116ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18062024-10-01T02:36:12.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95380 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46506 req_id = e44a1e58-ac36-48a8-8cf7-ff06e30c55ce response_code = 204 uri = /switch-ports
18072024-10-01T02:36:12.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46506 req_id = 5a351f45-c90f-4d35-8469-73b712fbb4da response_code = 200 uri = /network-bootstore-config
18082024-10-01T02:36:16.192ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33544
18092024-10-01T02:36:16.192ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 75 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33544 req_id = a8b9269f-22d8-44b9-a58d-25e6726b5f91 response_code = 200 uri = /baseboard
18102024-10-01T02:36:22.416ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8584 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 9336a5d4-19a4-4009-9963-22ac8f1dc61e response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
18112024-10-01T02:36:41.286ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37088
18122024-10-01T02:36:41.286ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37088 req_id = 457e29f8-d14b-4788-a9b5-06faf2c5000a response_code = 200 uri = /vpc-routes
18132024-10-01T02:36:41.303ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61722
18142024-10-01T02:36:41.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61722 req_id = d20e261f-815f-41c5-b2c5-9e19daee8cf5 response_code = 200 uri = /vpc-routes
18152024-10-01T02:36:41.311ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43063
18162024-10-01T02:36:41.311ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43063 req_id = ecee5f94-f99e-4b85-b4ee-9c2c6ede361e response_code = 200 uri = /v2p
18172024-10-01T02:36:41.315ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 871 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61722 req_id = e3a4d2e2-9af2-4728-8e3f-adc8ba383a6f response_code = 204 uri = /vpc-routes
18182024-10-01T02:36:41.315ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37088 req_id = 8b6f4528-accb-4f75-a73f-964f117a27ca response_code = 204 uri = /vpc-routes
18192024-10-01T02:36:41.347ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62423
18202024-10-01T02:36:41.348ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62423 req_id = 0361b053-b245-4129-948c-af5022dbda93 response_code = 200 uri = /v2p
18212024-10-01T02:36:41.424ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38727
18222024-10-01T02:36:41.424ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18232024-10-01T02:36:41.473ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18242024-10-01T02:36:41.561ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 137008 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38727 req_id = 915e0578-e48e-45ca-b039-291fbd1d79f5 response_code = 204 uri = /switch-ports
18252024-10-01T02:36:41.561ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44576
18262024-10-01T02:36:41.561ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 296 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38727 req_id = 31629ace-4db2-4ac7-a34b-98ec2a9e3d0a response_code = 200 uri = /network-bootstore-config
18272024-10-01T02:36:41.562ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18282024-10-01T02:36:41.621ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18292024-10-01T02:36:41.686ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45044
18302024-10-01T02:36:41.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 832 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45044 req_id = 4f35ad07-0372-46c6-8e4b-1f66ffc179ec response_code = 200 uri = /vpc-routes
18312024-10-01T02:36:41.698ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45044 req_id = 3bcd1b89-3126-46d4-9100-8f8d65e35bb7 response_code = 204 uri = /vpc-routes
18322024-10-01T02:36:41.704ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 143177 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44576 req_id = 95ae5f2c-4372-4760-85b4-b585e762a37a response_code = 204 uri = /switch-ports
18332024-10-01T02:36:41.704ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44576 req_id = 40091ca0-afda-49f5-8c45-f3335a3419fd response_code = 200 uri = /network-bootstore-config
18342024-10-01T02:36:41.752ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64877
18352024-10-01T02:36:41.752ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18362024-10-01T02:36:41.803ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18372024-10-01T02:36:41.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129332 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64877 req_id = 9a4358a5-3d70-48b3-bade-b2835927d7f3 response_code = 204 uri = /switch-ports
18382024-10-01T02:36:41.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38321
18392024-10-01T02:36:41.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64877 req_id = 7876993f-0b60-4d58-994e-8796cda80fb8 response_code = 200 uri = /network-bootstore-config
18402024-10-01T02:36:41.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 235 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38321 req_id = fc1ff5af-6666-4a34-9731-2c9070921557 response_code = 200 uri = /v2p
18412024-10-01T02:36:46.226ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52505
18422024-10-01T02:36:46.227ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 601 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52505 req_id = ab303412-66f4-441c-9e7e-e1fca65173ae response_code = 200 uri = /baseboard
18432024-10-01T02:36:52.417ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8875 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 960feb40-0b0f-40db-ab2e-a7f73d7174fb response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
18442024-10-01T02:37:11.406ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41015
18452024-10-01T02:37:11.406ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41015 req_id = 36c23e75-455e-4c19-a0bf-7e107046a29d response_code = 200 uri = /vpc-routes
18462024-10-01T02:37:11.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41015 req_id = 42fa2950-e5bf-48ce-a5c1-a0f7d91d97a8 response_code = 204 uri = /vpc-routes
18472024-10-01T02:37:11.479ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40004
18482024-10-01T02:37:11.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 924 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40004 req_id = 17912703-d013-46e1-9415-1045175ebdb1 response_code = 200 uri = /vpc-routes
18492024-10-01T02:37:11.494ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1089 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40004 req_id = c639be97-8d2d-4e74-b829-f36d21867797 response_code = 204 uri = /vpc-routes
18502024-10-01T02:37:11.577ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59320
18512024-10-01T02:37:11.577ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18522024-10-01T02:37:11.620ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18532024-10-01T02:37:11.682ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 105586 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59320 req_id = 2eb88e5a-7305-4e40-8d7e-ba17b5789e5d response_code = 204 uri = /switch-ports
18542024-10-01T02:37:11.683ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62041
18552024-10-01T02:37:11.683ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40326
18562024-10-01T02:37:11.683ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18572024-10-01T02:37:11.683ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 256 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59320 req_id = ee8b972a-058e-463b-a1cf-46af51dedb2d response_code = 200 uri = /network-bootstore-config
18582024-10-01T02:37:11.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3912 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40326 req_id = 0719290e-33d7-41d8-96db-bca964a90e09 response_code = 200 uri = /v2p
18592024-10-01T02:37:11.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37730
18602024-10-01T02:37:11.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 970 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37730 req_id = 4879ed63-e215-4175-883e-970bb7ecbc7d response_code = 200 uri = /v2p
18612024-10-01T02:37:11.718ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18622024-10-01T02:37:11.739ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35313
18632024-10-01T02:37:11.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35313 req_id = 1c6d2afb-d92d-41bb-8824-184dc40caacc response_code = 200 uri = /vpc-routes
18642024-10-01T02:37:11.755ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 166 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35313 req_id = 1e059471-b7bb-435e-8768-892a0794a208 response_code = 204 uri = /vpc-routes
18652024-10-01T02:37:11.789ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 105924 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62041 req_id = 5012be57-14cc-4bf6-991d-8276b6f70582 response_code = 204 uri = /switch-ports
18662024-10-01T02:37:11.791ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1566 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62041 req_id = 4c94e7d1-aaba-4e43-a6e6-205ad8fa9e15 response_code = 200 uri = /network-bootstore-config
18672024-10-01T02:37:12.125ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42527
18682024-10-01T02:37:12.125ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42527 req_id = c3a6b84f-b37c-4809-9a79-5289c08ed972 response_code = 200 uri = /v2p
18692024-10-01T02:37:12.182ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49283
18702024-10-01T02:37:12.182ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18712024-10-01T02:37:12.222ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18722024-10-01T02:37:12.281ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99058 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49283 req_id = 2208c0ea-bd13-4362-bc82-9289f041cdf4 response_code = 204 uri = /switch-ports
18732024-10-01T02:37:12.281ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49283 req_id = 776d3ba9-0917-47da-bdd3-da4dc2233cd5 response_code = 200 uri = /network-bootstore-config
18742024-10-01T02:37:16.261ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43086
18752024-10-01T02:37:16.261ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 68 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43086 req_id = 4442f181-5734-4e29-9701-8a17a4d0fb3c response_code = 200 uri = /baseboard
18762024-10-01T02:37:22.417ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8505 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 4f00f6be-33c0-4183-8766-219210892b7c response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
18772024-10-01T02:37:41.195ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51377
18782024-10-01T02:37:41.196ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51377 req_id = c83b3397-4307-4dd2-ae06-3b2ea60a3385 response_code = 200 uri = /vpc-routes
18792024-10-01T02:37:41.209ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51377 req_id = a2afa48b-cb28-4171-954b-7ebbee2f9752 response_code = 204 uri = /vpc-routes
18802024-10-01T02:37:41.274ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41166
18812024-10-01T02:37:41.274ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 31 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41166 req_id = c3781555-e68f-4c90-bb90-82c9ba8897cb response_code = 200 uri = /vpc-routes
18822024-10-01T02:37:41.284ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41166 req_id = 3bbf05a0-e37c-4e91-a021-816a14d4d3b6 response_code = 204 uri = /vpc-routes
18832024-10-01T02:37:41.350ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58537
18842024-10-01T02:37:41.350ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18852024-10-01T02:37:41.389ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18862024-10-01T02:37:41.446ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95808 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58537 req_id = 06dcd33f-e7fc-44e5-aced-e7622a6d4907 response_code = 204 uri = /switch-ports
18872024-10-01T02:37:41.446ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60149
18882024-10-01T02:37:41.446ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58645
18892024-10-01T02:37:41.446ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 245 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58645 req_id = 46fce393-96ea-4e17-a00f-cbda5689e3d6 response_code = 200 uri = /v2p
18902024-10-01T02:37:41.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 294 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58537 req_id = 0a6a4850-0ce4-4758-8bec-29f3b8a49c1a response_code = 200 uri = /network-bootstore-config
18912024-10-01T02:37:41.447ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
18922024-10-01T02:37:41.490ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
18932024-10-01T02:37:41.520ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62119
18942024-10-01T02:37:41.520ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62119 req_id = fcde57fa-a4e1-4f07-bc8d-e6991ddebb4c response_code = 200 uri = /v2p
18952024-10-01T02:37:41.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129563 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60149 req_id = 1169f2d0-2482-4628-97a0-cad6bfaef708 response_code = 204 uri = /switch-ports
18962024-10-01T02:37:41.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 196 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60149 req_id = 395281fd-e065-4466-8c5d-bc65b2b1a2e7 response_code = 200 uri = /network-bootstore-config
18972024-10-01T02:37:41.696ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57280
18982024-10-01T02:37:41.696ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57280 req_id = 1cb9668b-85d4-4c81-adef-64c5b6b82202 response_code = 200 uri = /vpc-routes
18992024-10-01T02:37:41.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57280 req_id = ff842039-e67d-4341-ae4d-7f99eb86c199 response_code = 204 uri = /vpc-routes
19002024-10-01T02:37:41.764ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45129
19012024-10-01T02:37:41.764ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45129 req_id = 053cef42-c53e-403c-b28b-6cd3b8613cc8 response_code = 200 uri = /v2p
19022024-10-01T02:37:41.808ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64537
19032024-10-01T02:37:41.808ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
19042024-10-01T02:37:41.852ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
19052024-10-01T02:37:41.907ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99355 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64537 req_id = de06ff0d-3c10-485e-96df-93bea964f359 response_code = 204 uri = /switch-ports
19062024-10-01T02:37:41.907ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64537 req_id = 6a25f6f0-1ea8-4343-b9d0-277e283efba2 response_code = 200 uri = /network-bootstore-config
19072024-10-01T02:37:46.294ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44743
19082024-10-01T02:37:46.294ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 75 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44743 req_id = 651db12b-9642-4c2c-8728-d2963a701c22 response_code = 200 uri = /baseboard
19092024-10-01T02:37:52.418ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9150 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 34f2ca3e-7bc2-4eb3-b0d4-0512feaf1ad0 response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
19102024-10-01T02:38:11.370ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49875
19112024-10-01T02:38:11.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49875 req_id = 5c3ff38c-2e0f-40f0-a5fd-c0b2226049ca response_code = 200 uri = /vpc-routes
19122024-10-01T02:38:11.381ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49875 req_id = a96dde67-43c1-4734-a9ba-17e6064f74b7 response_code = 204 uri = /vpc-routes
19132024-10-01T02:38:11.430ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59229
19142024-10-01T02:38:11.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59229 req_id = f91bc20c-b1be-4d49-b276-9610502b2b22 response_code = 200 uri = /vpc-routes
19152024-10-01T02:38:11.451ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 163 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59229 req_id = bc364c39-4cb8-42ee-9ad4-b43b1dbaa5b5 response_code = 204 uri = /vpc-routes
19162024-10-01T02:38:11.551ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64855
19172024-10-01T02:38:11.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1825 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64855 req_id = 2c39033b-b309-4d43-b17b-0f7bc2487657 response_code = 200 uri = /v2p
19182024-10-01T02:38:11.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51477
19192024-10-01T02:38:11.628ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
19202024-10-01T02:38:11.683ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
19212024-10-01T02:38:11.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129995 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51477 req_id = 72dc2361-972b-4be2-8a30-c555bd7923c5 response_code = 204 uri = /switch-ports
19222024-10-01T02:38:11.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60947
19232024-10-01T02:38:11.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57716
19242024-10-01T02:38:11.758ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
19252024-10-01T02:38:11.763ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4771 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51477 req_id = 2cf4a8ca-f3e4-4794-9f8f-f6c0d2dd7872 response_code = 200 uri = /network-bootstore-config
19262024-10-01T02:38:11.763ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4757 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60947 req_id = ac4d80ca-a84d-4d25-9c51-6c8393cc2156 response_code = 200 uri = /v2p
19272024-10-01T02:38:11.777ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38806
19282024-10-01T02:38:11.778ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
19292024-10-01T02:38:11.778ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
19302024-10-01T02:38:11.778ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19312024-10-01T02:38:11.778ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19322024-10-01T02:38:11.778ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19332024-10-01T02:38:11.778ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19342024-10-01T02:38:11.778ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19352024-10-01T02:38:11.778ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 444 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38806 req_id = 2aca0bc6-0001-43e0-897b-4cb315dc92a8 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
19362024-10-01T02:38:11.793ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
19372024-10-01T02:38:11.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99157 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57716 req_id = 30c85455-6d0b-4876-8d6e-00d69a0db16a response_code = 204 uri = /switch-ports
19382024-10-01T02:38:11.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57716 req_id = 97b615ab-7070-4933-8e33-2dc9309d4c92 response_code = 200 uri = /network-bootstore-config
19392024-10-01T02:38:11.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47815
19402024-10-01T02:38:11.897ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
19412024-10-01T02:38:11.897ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
19422024-10-01T02:38:11.897ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19432024-10-01T02:38:11.897ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19442024-10-01T02:38:11.897ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19452024-10-01T02:38:11.897ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19462024-10-01T02:38:11.897ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19472024-10-01T02:38:11.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1018 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47815 req_id = a9bdc2f7-fa79-42e1-a1b0-42d8cfe6a378 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
19482024-10-01T02:38:12.063ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34797
19492024-10-01T02:38:12.063ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34797 req_id = a4d63860-8601-43e8-b938-1c841265b1f8 response_code = 200 uri = /vpc-routes
19502024-10-01T02:38:12.117ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57479
19512024-10-01T02:38:12.117ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57479 req_id = 8c290063-ffaa-489b-8550-2680c313c108 response_code = 200 uri = /v2p
19522024-10-01T02:38:12.126ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34797 req_id = 68afde62-c7a4-4708-95d1-5e671e67f8c8 response_code = 204 uri = /vpc-routes
19532024-10-01T02:38:12.281ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46346
19542024-10-01T02:38:12.281ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 498a9376-2ad6-4307-b191-e6d73cf2f1a4, kind: Service { id: 0a1ed896-5078-4267-b636-10151376f015 }, name: Name("external-dns-0a1ed896-5078-4267-b636-10151376f015"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 226, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e4ffe935-3383-46e7-947f-e87df509f06f, kind: Service { id: eb937866-c779-4a9e-aec4-cc18302d4ce1 }, name: Name("external-dns-eb937866-c779-4a9e-aec4-cc18302d4ce1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 225])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 611bfb6e-9731-4e0b-9af8-03e07b661aa3, kind: Service { id: 772acfa5-ee20-4a17-a053-c207ffa50ed6 }, name: Name("nexus-772acfa5-ee20-4a17-a053-c207ffa50ed6"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 222])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 98b85cfc-ac80-4175-9f04-dd15c9b20d24, kind: Service { id: 3dfd7033-71d8-4eb9-bb97-49dae3b5cd35 }, name: Name("nexus-3dfd7033-71d8-4eb9-bb97-49dae3b5cd35"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 167, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a840a1d8-f258-45fe-96a0-7b8383461d51, kind: Service { id: e9e4c757-e076-4386-92e2-8217b8bb307d }, name: Name("nexus-e9e4c757-e076-4386-92e2-8217b8bb307d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 176, 241])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
19552024-10-01T02:38:12.281ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
19562024-10-01T02:38:12.281ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19572024-10-01T02:38:12.282ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19582024-10-01T02:38:12.282ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19592024-10-01T02:38:12.282ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
19602024-10-01T02:38:12.282ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
19612024-10-01T02:38:12.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 303 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46346 req_id = a7a85127-0774-4c79-a7c3-35dcce6c162a response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
19622024-10-01T02:38:12.295ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49911
19632024-10-01T02:38:12.296ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
19642024-10-01T02:38:12.333ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
19652024-10-01T02:38:12.385ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89804 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49911 req_id = 3cbd132e-ce3b-4864-9236-819d6da34c82 response_code = 204 uri = /switch-ports
19662024-10-01T02:38:12.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49911 req_id = 694e1766-ddd6-43f1-8325-baa286c5a857 response_code = 200 uri = /network-bootstore-config
19672024-10-01T02:38:16.326ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60696
19682024-10-01T02:38:16.326ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60696 req_id = 3d75aebd-6a37-4bfc-90cd-a4a25ba5cf28 response_code = 200 uri = /baseboard
19692024-10-01T02:38:22.423ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 11532 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 8ec2d74c-5a59-4d01-ac1d-c2ead9101f24 response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
19702024-10-01T02:38:41.242ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41076
19712024-10-01T02:38:41.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41076 req_id = a63faeed-486b-4ae9-8831-35f1c8cd2c08 response_code = 200 uri = /vpc-routes
19722024-10-01T02:38:41.251ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41076 req_id = 1bd60b4a-53c8-4868-af3b-a6f3f2b54974 response_code = 204 uri = /vpc-routes
19732024-10-01T02:38:41.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49342
19742024-10-01T02:38:41.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 31 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49342 req_id = 3f763a4f-3aae-45fd-b109-4f3d12f8aa9d response_code = 200 uri = /vpc-routes
19752024-10-01T02:38:41.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 479 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49342 req_id = 01b2408c-2910-4449-b34c-2741ea4d60d4 response_code = 204 uri = /vpc-routes
19762024-10-01T02:38:41.311ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53356
19772024-10-01T02:38:41.312ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 607 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53356 req_id = 288e10e6-cc0a-4962-a88d-d37c3ceae58d response_code = 200 uri = /v2p
19782024-10-01T02:38:41.334ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64170
19792024-10-01T02:38:41.335ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64170 req_id = a1a3c6b2-2483-4055-bd3b-91e22af3788b response_code = 200 uri = /v2p
19802024-10-01T02:38:41.367ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63304
19812024-10-01T02:38:41.367ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
19822024-10-01T02:38:41.407ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
19832024-10-01T02:38:41.462ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95487 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63304 req_id = de4d7c5e-35e3-40e4-8d25-bdbdeae97bca response_code = 204 uri = /switch-ports
19842024-10-01T02:38:41.462ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40469
19852024-10-01T02:38:41.463ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
19862024-10-01T02:38:41.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3638 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63304 req_id = 97e426e0-1e26-4443-9001-6b7d99b06ed9 response_code = 200 uri = /network-bootstore-config
19872024-10-01T02:38:41.501ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
19882024-10-01T02:38:41.584ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121748 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40469 req_id = 8c15513f-5495-4b80-baf4-36de69cb128c response_code = 204 uri = /switch-ports
19892024-10-01T02:38:41.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40469 req_id = 3020f0dc-c034-429e-90fe-3b8cab534d57 response_code = 200 uri = /network-bootstore-config
19902024-10-01T02:38:41.846ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37031
19912024-10-01T02:38:41.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37031 req_id = 6a83407e-e4dd-406f-9064-dada719c62fe response_code = 200 uri = /vpc-routes
19922024-10-01T02:38:41.964ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63270
19932024-10-01T02:38:41.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63270 req_id = 94dd01f1-560e-4d83-8a3b-c9aba194abf6 response_code = 200 uri = /v2p
19942024-10-01T02:38:41.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37031 req_id = 45f4a632-5433-413e-b660-232baae9a893 response_code = 204 uri = /vpc-routes
19952024-10-01T02:38:42.023ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55871
19962024-10-01T02:38:42.023ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
19972024-10-01T02:38:42.064ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
19982024-10-01T02:38:42.117ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93627 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:55871 req_id = f3f97869-d671-4b75-a1a3-9956a017d9bb response_code = 204 uri = /switch-ports
19992024-10-01T02:38:42.117ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 656 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55871 req_id = 7488cfab-9d74-43be-9b0b-a05f2811bdab response_code = 200 uri = /network-bootstore-config
20002024-10-01T02:38:46.358ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61235
20012024-10-01T02:38:46.358ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61235 req_id = 689cc54e-673f-425c-9cbb-9c9866e6d546 response_code = 200 uri = /baseboard
20022024-10-01T02:38:52.499ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7772 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 7f1dd5d9-119d-4860-8d59-f91bbf72b0ea response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
20032024-10-01T02:39:11.438ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36489
20042024-10-01T02:39:11.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 930 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36489 req_id = dd0fed54-f704-4857-b769-fb161ecfda12 response_code = 200 uri = /vpc-routes
20052024-10-01T02:39:11.584ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44348
20062024-10-01T02:39:11.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44348 req_id = b7e2d7cc-af2c-42f1-abb0-f38e744559e4 response_code = 200 uri = /vpc-routes
20072024-10-01T02:39:11.587ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48180
20082024-10-01T02:39:11.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 200 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48180 req_id = 6d30fd2e-66a4-4cc9-9aff-5d22920de512 response_code = 200 uri = /v2p
20092024-10-01T02:39:11.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 236 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36489 req_id = f7bcb442-d86e-496f-ac5f-fcbe12ed58fd response_code = 204 uri = /vpc-routes
20102024-10-01T02:39:11.607ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44348 req_id = d43c73d3-ded4-45cb-843a-ac814f2b94e4 response_code = 204 uri = /vpc-routes
20112024-10-01T02:39:11.684ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36893
20122024-10-01T02:39:11.686ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
20132024-10-01T02:39:11.724ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
20142024-10-01T02:39:11.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97114 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36893 req_id = 5bb8efc1-7885-463a-ad8e-b9ea6daf2c4f response_code = 204 uri = /switch-ports
20152024-10-01T02:39:11.783ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63963
20162024-10-01T02:39:11.783ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55395
20172024-10-01T02:39:11.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 214 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55395 req_id = 72bd510e-a9f0-438d-b5a8-af0b960e4df1 response_code = 200 uri = /v2p
20182024-10-01T02:39:11.783ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
20192024-10-01T02:39:11.803ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 271 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36893 req_id = c4a97c03-3f53-4fc6-83be-b726505669c3 response_code = 200 uri = /network-bootstore-config
20202024-10-01T02:39:11.822ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
20212024-10-01T02:39:11.853ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56304
20222024-10-01T02:39:11.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56304 req_id = 51cc3bd1-e0b6-455f-868c-7920077c0fb9 response_code = 200 uri = /vpc-routes
20232024-10-01T02:39:11.863ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 266 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56304 req_id = a655fd13-0a32-4d6b-9edf-be97eac0b7e8 response_code = 204 uri = /vpc-routes
20242024-10-01T02:39:11.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99200 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63963 req_id = f2e0a7a2-e4dc-4e0d-94e4-fb3ed8242596 response_code = 204 uri = /switch-ports
20252024-10-01T02:39:11.885ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 901 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63963 req_id = 2fda1a3c-764c-43a3-803e-d6fe7af3ef47 response_code = 200 uri = /network-bootstore-config
20262024-10-01T02:39:11.893ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34793
20272024-10-01T02:39:11.893ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34793 req_id = 12bf1aa2-41c7-44e1-88d4-7920bf106378 response_code = 200 uri = /v2p
20282024-10-01T02:39:12.085ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53534
20292024-10-01T02:39:12.086ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
20302024-10-01T02:39:12.124ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
20312024-10-01T02:39:12.181ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95923 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53534 req_id = 5777ddd7-e9e8-4301-ba54-90d3ae256754 response_code = 204 uri = /switch-ports
20322024-10-01T02:39:12.182ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53534 req_id = a9fee1f9-fff2-4fd4-a1d0-98db023f1bf7 response_code = 200 uri = /network-bootstore-config
20332024-10-01T02:39:16.393ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63883
20342024-10-01T02:39:16.393ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 489 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63883 req_id = 9957f3e7-90cc-4045-ad65-6b23b6f1a189 response_code = 200 uri = /baseboard
20352024-10-01T02:39:22.500ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9052 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = 38d82a37-5063-4eec-be3d-e20bc45703d1 response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
20362024-10-01T02:39:39.304ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46027
20372024-10-01T02:39:39.305ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 193 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46027 req_id = 41f0e326-76a4-414d-9742-13d8c77b970c response_code = 200 uri = /v2p
20382024-10-01T02:39:39.306ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(12861255) }
20392024-10-01T02:39:39.306ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46027 req_id = 9d5cf2eb-35a5-4127-b81e-66ebd6dabd0b response_code = 204 uri = /v2p
20402024-10-01T02:39:39.579ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56298
20412024-10-01T02:39:39.591ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:539 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(12861255))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-826a7ad9c27", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: e9941210-4ee5-4073-bb20-cb2cc34f87e8, read_only_parent: Some(Volume { block_size: 512, id: 43090e20-41aa-49b6-838a-60b2e7cf58b9, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 8eb66a65-8238-4ec3-a450-6d88e1bd6af4, key: Some("HzsEC8bGbGH/SU+5qxAPLgqhAegtjsVdTkbEsSV/HTA="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: e9941210-4ee5-4073-bb20-cb2cc34f87e8, key: Some("otPgSRdW7jr6vr8jqjfITVaNwbkLEmeHHwFPhuSuoho="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::15]:19003", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::12]:19002"] } }] } }], cloud_init_bytes: Some(..) } instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 metadata = InstanceMetadata { silo_id: 21983b87-d6d0-47c6-a33c-5fe68fbed0c6, project_id: ade3b588-424d-4931-8c7d-7d3a43246cab } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-01T02:39:39.578732322Z }
20422024-10-01T02:39:39.591ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:572 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 migration_id = None propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
20432024-10-01T02:39:39.592ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1008 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 migration_id = None propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(12861255))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 86681bd0-1b49-4295-a785-f836a4a342ae, kind: Instance { id: 155a6d94-b6d1-469e-afb5-9a278b60eab5 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12861255), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-826a7ad9c27", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: e9941210-4ee5-4073-bb20-cb2cc34f87e8, read_only_parent: Some(Volume { block_size: 512, id: 43090e20-41aa-49b6-838a-60b2e7cf58b9, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 8eb66a65-8238-4ec3-a450-6d88e1bd6af4, key: Some("HzsEC8bGbGH/SU+5qxAPLgqhAegtjsVdTkbEsSV/HTA="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: e9941210-4ee5-4073-bb20-cb2cc34f87e8, key: Some("otPgSRdW7jr6vr8jqjfITVaNwbkLEmeHHwFPhuSuoho="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::15]:19003", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::12]:19002"] } }] } }], cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-01T02:39:39.578732322Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
20442024-10-01T02:39:39.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 16942 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56298 req_id = f20421f2-c9ee-46d4-ab65-10d375990127 response_code = 200 uri = /vmms/031b6427-1460-4676-a469-9de2f0f9317d
20452024-10-01T02:39:39.673ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38205
20462024-10-01T02:39:39.673ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 297 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38205 req_id = 25fe09e7-b974-4ab5-8b18-a41184399f92 response_code = 200 uri = /vmms/031b6427-1460-4676-a469-9de2f0f9317d/state
20472024-10-01T02:39:39.675ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 12861255 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
20482024-10-01T02:39:39.676ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20492024-10-01T02:39:39.715ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d file = illumos-utils/src/zone.rs:316 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
20502024-10-01T02:39:39.736ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d file = illumos-utils/src/zone.rs:354 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
20512024-10-01T02:39:41.308ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58799
20522024-10-01T02:39:41.319ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52753
20532024-10-01T02:39:41.358ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58603
20542024-10-01T02:39:41.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 59 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58603 req_id = 15c69b96-1282-4713-8369-7283460fe351 response_code = 200 uri = /vpc-routes
20552024-10-01T02:39:41.379ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
20562024-10-01T02:39:41.379ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d zone file = illumos-utils/src/running_zone.rs:488 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20572024-10-01T02:39:41.440ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60020
20582024-10-01T02:39:41.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60020 req_id = 62e06936-b725-4ea5-af0f-4794f0d6a55b response_code = 200 uri = /vpc-routes
20592024-10-01T02:39:41.472ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57784
20602024-10-01T02:39:41.472ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 204 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57784 req_id = 528fc284-c8e2-4c6f-abde-1cc50bcb0c8b response_code = 200 uri = /v2p
20612024-10-01T02:39:41.481ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51518
20622024-10-01T02:39:41.481ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
20632024-10-01T02:39:41.545ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
20642024-10-01T02:39:41.640ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 158437 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51518 req_id = e4725d2c-1716-47f9-bfe9-174c8ad63938 response_code = 204 uri = /switch-ports
20652024-10-01T02:39:41.640ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45868
20662024-10-01T02:39:41.640ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40125
20672024-10-01T02:39:41.641ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
20682024-10-01T02:39:41.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 5918 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40125 req_id = 6bf37160-0840-495d-82ed-abf58f1b64cf response_code = 200 uri = /v2p
20692024-10-01T02:39:41.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6289 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60020 req_id = 0f76c9b6-facf-43ef-9a4a-daadeeb84957 response_code = 204 uri = /vpc-routes
20702024-10-01T02:39:41.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6450 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58603 req_id = 0412d8d5-6f49-47da-9e71-ddd9aadb0ab6 response_code = 204 uri = /vpc-routes
20712024-10-01T02:39:41.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6351 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51518 req_id = 57031c66-dfe6-4b16-a944-7152b3332562 response_code = 200 uri = /network-bootstore-config
20722024-10-01T02:39:41.661ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 38.91601ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20732024-10-01T02:39:41.691ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
20742024-10-01T02:39:41.714ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 71.734415ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20752024-10-01T02:39:41.800ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 159884 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45868 req_id = 435f3b9c-e96f-4701-b7a0-476fd9f37554 response_code = 204 uri = /switch-ports
20762024-10-01T02:39:41.803ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2460 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45868 req_id = 0671488d-fbfc-449e-a861-9bea05254a27 response_code = 200 uri = /network-bootstore-config
20772024-10-01T02:39:41.805ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 189.891774ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20782024-10-01T02:39:41.807ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48053
20792024-10-01T02:39:41.835ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55743
20802024-10-01T02:39:41.835ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55743 req_id = 2407231d-6bf1-4902-863b-f738a33263ef response_code = 200 uri = /vpc-routes
20812024-10-01T02:39:41.885ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55743 req_id = 91a6f594-b510-45f4-9864-3a6d4ace98cc response_code = 204 uri = /vpc-routes
20822024-10-01T02:39:41.967ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50966
20832024-10-01T02:39:41.967ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4087
20842024-10-01T02:39:42.018ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4111
20852024-10-01T02:39:42.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 136584 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50966 req_id = 45b9cc65-0fbf-44df-bc2b-99772defe88d response_code = 204 uri = /switch-ports
20862024-10-01T02:39:42.108ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64888
20872024-10-01T02:39:42.108ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1326 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50966 req_id = a11b94c0-223a-49a7-b8b2-07549822efb0 response_code = 200 uri = /network-bootstore-config
20882024-10-01T02:39:42.108ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 474 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64888 req_id = 889a2eb6-0ce4-4bdd-8453-e70d7465ae18 response_code = 200 uri = /v2p
20892024-10-01T02:39:42.118ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 469.899666ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20902024-10-01T02:39:42.599ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 1.165746838s file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20912024-10-01T02:39:43.777ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 1.109222118s file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20922024-10-01T02:39:44.897ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 1.017881055s file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20932024-10-01T02:39:45.925ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 934.904452ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d zone = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
20942024-10-01T02:39:46.427ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56888
20952024-10-01T02:39:46.427ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56888 req_id = a44183f6-a868-4934-80f5-91fdb3c681a3 response_code = 200 uri = /baseboard
20962024-10-01T02:39:46.889ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d file = sled-agent/src/instance.rs:1466 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
20972024-10-01T02:39:46.899ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 43.30672ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
20982024-10-01T02:39:46.953ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 89.271563ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
20992024-10-01T02:39:47.052ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 205.799834ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21002024-10-01T02:39:47.269ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d") failed: Property not found. retry in 516.603407ms file = illumos-utils/src/svc.rs:36 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21012024-10-01T02:39:47.799ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1475 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21022024-10-01T02:39:47.837ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1499 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21032024-10-01T02:39:47.979ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:744 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21042024-10-01T02:39:47.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6671992 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58799 req_id = 1e3672bf-7ea7-4ed8-b42c-387c863716ab response_code = 200 uri = /vmms/031b6427-1460-4676-a469-9de2f0f9317d/state
21052024-10-01T02:39:47.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6172831 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48053 req_id = 80213540-6b7f-41d6-b15f-6f042f60267e response_code = 200 uri = /vmms/031b6427-1460-4676-a469-9de2f0f9317d/state
21062024-10-01T02:39:47.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6661637 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52753 req_id = b7751c83-996a-47a8-b70e-fb17d10bb8a3 response_code = 200 uri = /vmms/031b6427-1460-4676-a469-9de2f0f9317d/state
21072024-10-01T02:39:47.981ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-10-01T02:39:47.981324803Z } file = sled-agent/src/instance.rs:594 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21082024-10-01T02:39:47.981ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-01T02:39:47.981324803Z } propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21092024-10-01T02:39:47.981ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-01T02:39:47.981324803Z }, migration_in: None, migration_out: None }
21102024-10-01T02:39:48.059ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-10-01T02:39:48.059406881Z } file = sled-agent/src/instance.rs:594 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21112024-10-01T02:39:48.059ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-01T02:39:48.059406881Z } propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21122024-10-01T02:39:48.059ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-01T02:39:48.059406881Z }, migration_in: None, migration_out: None }
21132024-10-01T02:39:52.502ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9823 local_addr = [fd00:1122:3344:101::1]:45292 method = GET remote_addr = [fd00:1122:3344:101::d]:60055 req_id = a206115d-8137-493b-a953-b13d340d95d5 response_code = 200 sled_id = 2a571ec0-85a3-450e-88fc-680c13ffa211 uri = /2a571ec0-85a3-450e-88fc-680c13ffa211
21142024-10-01T02:39:56.555ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62216
21152024-10-01T02:39:56.557ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1285 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62216 req_id = 69758af6-fcd2-4d47-979f-65298c06a391 response_code = 200 uri = /vmms/031b6427-1460-4676-a469-9de2f0f9317d/state
21162024-10-01T02:39:56.557ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-10-01T02:39:56.557325359Z } file = sled-agent/src/instance.rs:594 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21172024-10-01T02:39:56.557ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-01T02:39:56.557325359Z } propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21182024-10-01T02:39:56.557ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-01T02:39:56.557325359Z }, migration_in: None, migration_out: None }
21192024-10-01T02:39:56.613ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-10-01T02:39:56.613628592Z } file = sled-agent/src/instance.rs:594 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21202024-10-01T02:39:56.613ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-10-01T02:39:56.613628592Z } propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21212024-10-01T02:39:56.613ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:630 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21222024-10-01T02:39:56.613ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21232024-10-01T02:39:56.618ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21242024-10-01T02:39:56.618ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/09a98b58-1585-44a1-a9f9-fa7c634d7922/debug/bundle/zone", "/pool/int/1586f214-1a8d-4f2a-b9d3-907d2432a63f/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/52791279-0328-47f2-8641-652f421a0533/crypt/debug", "/pool/ext/924badcc-07d3-4192-87c0-257ed1dc2017/crypt/debug", "/pool/ext/ff4b0e2e-bb96-457c-a345-9f8b3478a5a8/crypt/debug", "/pool/ext/efd83ea1-94bf-48aa-a059-ed387b79a77f/crypt/debug", "/pool/ext/a0b3d8f3-189e-4ea5-a742-9dc63f4ac87d/crypt/debug", "/pool/ext/4b58fe92-3fb4-46b4-9ea6-d620062bf2a1/crypt/debug", "/pool/ext/ee22443f-adfe-4fb0-bef1-d9b053b6a9d8/crypt/debug", "/pool/ext/a04e4a42-059d-4585-a24d-8ec28aa914e5/crypt/debug", "/pool/ext/841bd0ff-a375-452e-9289-601e29937aad/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d
21252024-10-01T02:39:57.269ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d", bundle_id: e04caaba-97c9-40b6-a823-0f0c5429ae36 }, time_created: 2024-10-01T02:39:56.619450020Z, version: 0, cause: TerminatedInstance }
21262024-10-01T02:39:57.269ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_031b6427-1460-4676-a469-9de2f0f9317d file = sled-agent/src/instance.rs:843 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21272024-10-01T02:39:59.009ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21282024-10-01T02:39:59.026ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(12861255), kind: System }
21292024-10-01T02:39:59.026ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(12861255), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
21302024-10-01T02:39:59.031ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-01T02:39:59.031013960Z }, migration_in: None, migration_out: None }
21312024-10-01T02:39:59.085ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-01T02:39:59.031013960Z }, migration_in: None, migration_out: None }
21322024-10-01T02:39:59.089ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:780 instance_id = 155a6d94-b6d1-469e-afb5-9a278b60eab5 propolis_id = 031b6427-1460-4676-a469-9de2f0f9317d
21332024-10-01T02:39:59.633ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47592
21342024-10-01T02:39:59.633ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47592 req_id = ee3157d3-43d5-4f02-a8b3-2dcbeb244850 response_code = 200 uri = /vpc-routes
21352024-10-01T02:39:59.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47592 req_id = 6840a92b-068b-4b43-90af-f7d812665df4 response_code = 204 uri = /vpc-routes
21362024-10-01T02:39:59.684ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37614
21372024-10-01T02:39:59.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37614 req_id = 3956d17e-4d8c-4a52-a069-73c02abf020e response_code = 200 uri = /v2p
21382024-10-01T02:39:59.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35178
21392024-10-01T02:39:59.978ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 170 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35178 req_id = 52bd9bfd-b54e-475b-b970-33282ace7161 response_code = 200 uri = /v2p
21402024-10-01T02:40:00.883ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54737
21412024-10-01T02:40:00.885ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 965 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54737 req_id = 20c509e3-0f6b-4f29-8cd0-c7afd5e6a937 response_code = 200 uri = /vpc-routes
21422024-10-01T02:40:00.893ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54737 req_id = ca5d2087-b891-4ef3-b640-2f5e1cb44271 response_code = 204 uri = /vpc-routes