Max level shown:
1 [ Oct 9 05:47:49 Enabled. ]
2 [ Oct 9 05:47:49 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Oct 9 05:47:49 Method "start" exited with status 0. ]
4 [ Oct 9 05:47:49 Rereading configuration. ]
5 [ Oct 9 05:47:49 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-10-09T05:47:49.972ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-10-09T05:47:49.972ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-10-09T05:47:50.029ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-10-09T05:47:50.029ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-10-09T05:47:50.144ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-10-09T05:47:50.153ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-10-09T05:47:50.161ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-10-09T05:47:50.765ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-10-09T05:47:50.904ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-10-09T05:47:50.904ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-10-09T05:47:50.904ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-10-09T05:47:50.905ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-10-09T05:47:50.905ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-10-09T05:47:50.905ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-10-09T05:47:50.907ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-10-09T05:47:50.907ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-10-09T05:47:50.907ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-10-09T05:47:50.907ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-10-09T05:47:50.907ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-10-09T05:47:50.907ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-10-09T05:47:50.908ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-10-09T05:47:50.908ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-10-09T05:47:50.908ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-10-09T05:47:50.918ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-10-09T05:47:50.918ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool)
322024-10-09T05:47:51.015ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal }
332024-10-09T05:47:51.395ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-10-09T05:47:51.395ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-10-09T05:47:51.395ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-10-09T05:47:51.395ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-10-09T05:47:51.395ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-10-09T05:47:51.395ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-10-09T05:47:51.395ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-10-09T05:47:51.404ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412024-10-09T05:47:51.404ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool)
422024-10-09T05:47:51.488ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal }
432024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
532024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
542024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
602024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
632024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
642024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
652024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
662024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
692024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
702024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
712024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
722024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
732024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
782024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
822024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
832024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
842024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-10-09T05:47:51.895ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-10-09T05:47:51.895ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-10-09T05:47:51.895ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-10-09T05:47:51.895ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-10-09T05:47:51.895ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1002024-10-09T05:47:51.895ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1012024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1022024-10-09T05:47:51.895ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-10-09T05:47:51.895ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-10-09T05:47:51.895ZINFOSledAgent (bootstore): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-10-09T05:47:51.895ZINFOSledAgent (bootstore): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-10-09T05:47:51.895ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-10-09T05:47:51.895ZINFOSledAgent (bootstore): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-10-09T05:47:51.895ZINFOSledAgent (bootstore): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-10-09T05:47:51.895ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-10-09T05:47:51.895ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-10-09T05:47:51.895ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-10-09T05:47:51.906ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:739
1132024-10-09T05:47:51.906ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1342, tv_nsec: 188278416 } time_to_next_cleanup = 599.999933159s
1142024-10-09T05:47:51.906ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3944
1152024-10-09T05:47:51.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-10-09T05:47:51.908ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-10-09T05:47:51.908ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-10-09T05:47:51.909ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-10-09T05:47:51.910ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-10-09T05:47:51.911ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-10-09T05:47:51.911ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72
1222024-10-09T05:47:51.911ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-10-09T05:47:51.911ZINFOSledAgent (RSS): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-10-09T05:47:51.911ZINFOSledAgent (RSS): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-10-09T05:47:51.911ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-10-09T05:47:51.911ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-10-09T05:47:51.911ZINFOSledAgent (RSS): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-10-09T05:47:51.911ZINFOSledAgent (RSS): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-10-09T05:47:51.911ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-10-09T05:47:51.911ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-10-09T05:47:51.911ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-10-09T05:47:51.911ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-10-09T05:47:51.911ZINFOSledAgent (RSS): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-10-09T05:47:51.911ZINFOSledAgent (RSS): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-10-09T05:47:51.912ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: cde79127-ad1a-4dc8-8bde-c617fda5df2f, sleds: { [fdb0:18c0:4d0d:9fb2::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 8fa2b36a-fea8-4835-8541-6d298f48930d (sled), rack_id: cde79127-ad1a-4dc8-8bde-c617fda5df2f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-10-09T05:47:51.912ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-10-09T05:47:51.912ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-10-09T05:47:51.912ZINFOSledAgent (bootstore): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-10-09T05:47:51.912ZINFOSledAgent (bootstore): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-10-09T05:47:51.912ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-10-09T05:47:51.912ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 8fa2b36a-fea8-4835-8541-6d298f48930d (sled), rack_id: cde79127-ad1a-4dc8-8bde-c617fda5df2f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-10-09T05:47:51.913ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4241
1432024-10-09T05:47:51.913ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4623
1442024-10-09T05:47:51.917ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-10-09T05:47:51.923ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-10-09T05:47:51.923ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-10-09T05:47:51.927ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-10-09T05:47:51.927ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-10-09T05:47:51.927ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:37556
1502024-10-09T05:47:51.927ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 8fa2b36a-fea8-4835-8541-6d298f48930d (sled), rack_id: cde79127-ad1a-4dc8-8bde-c617fda5df2f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1512024-10-09T05:47:51.927ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1522024-10-09T05:47:51.927ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1532024-10-09T05:47:51.927ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72
1542024-10-09T05:47:51.927ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1552024-10-09T05:47:51.927ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1562024-10-09T05:47:51.927ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1572024-10-09T05:47:51.927ZINFOSledAgent (StorageManager): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1582024-10-09T05:47:51.927ZINFOSledAgent (StorageManager): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-10-09T05:47:51.927ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-10-09T05:47:51.927ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-10-09T05:47:51.927ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-10-09T05:47:51.927ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-10-09T05:47:51.928ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-10-09T05:47:51.944ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-10-09T05:47:51.963ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-10-09T05:47:51.980ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1672024-10-09T05:47:51.988ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
1682024-10-09T05:47:51.988ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1692024-10-09T05:47:51.988ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
1702024-10-09T05:47:51.988ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1712024-10-09T05:47:51.991ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 74.059582ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d3fd48fa-d48a-40d5-aa8d-1a3781dde8ce", "content-length": "124", "date": "Wed, 09 Oct 2024 05:47:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d3fd48fa-d48a-40d5-aa8d-1a3781dde8ce" } file = clients/ddm-admin-client/src/lib.rs:177
1722024-10-09T05:47:52.000ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_d6cde337-7634-4a9e-a6ab-bc764b121353/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-10-09T05:47:52.024ZINFOSledAgent: successfully created encrypted zvol: name="oxi_d6cde337-7634-4a9e-a6ab-bc764b121353/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-10-09T05:47:52.024ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_d6cde337-7634-4a9e-a6ab-bc764b121353/swap" file = sled-agent/src/swap_device.rs:96
1752024-10-09T05:47:52.029ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
1762024-10-09T05:47:52.029ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1772024-10-09T05:47:52.029ZINFOSledAgent (BackingFs): Ensuring dataset oxi_d6cde337-7634-4a9e-a6ab-bc764b121353/backing/fmd file = sled-agent/src/backing_fs.rs:138
1782024-10-09T05:47:52.066ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-10-09T05:47:52.067ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 102.817421ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "453fb51b-bd93-487c-948e-0d109f38e298", "content-length": "124", "date": "Wed, 09 Oct 2024 05:47:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "453fb51b-bd93-487c-948e-0d109f38e298" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-10-09T05:47:52.118ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1812024-10-09T05:47:52.171ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-10-09T05:47:52.172ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 91.091197ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b6a17f89-6fdd-4fbf-b779-c88b71012873", "content-length": "124", "date": "Wed, 09 Oct 2024 05:47:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b6a17f89-6fdd-4fbf-b779-c88b71012873" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-10-09T05:47:52.264ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-10-09T05:47:52.264ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 127.926066ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4b21a236-f157-4e3d-b564-ecdb1832141b", "content-length": "124", "date": "Wed, 09 Oct 2024 05:47:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4b21a236-f157-4e3d-b564-ecdb1832141b" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-10-09T05:47:52.336ZINFOSledAgent (BackingFs): Mounting oxi_d6cde337-7634-4a9e-a6ab-bc764b121353/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1862024-10-09T05:47:52.350ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1872024-10-09T05:47:52.350ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1882024-10-09T05:47:52.350ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1892024-10-09T05:47:52.350ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1902024-10-09T05:47:52.393ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-10-09T05:47:52.393ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 176.633181ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "878aca8a-8ed0-4d97-b518-3c47047809d5", "content-length": "124", "date": "Wed, 09 Oct 2024 05:47:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "878aca8a-8ed0-4d97-b518-3c47047809d5" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-10-09T05:47:52.571ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-10-09T05:47:52.572ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 333.540316ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "19042041-9e92-45e8-9764-cbddd88e30d3", "content-length": "124", "date": "Wed, 09 Oct 2024 05:47:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "19042041-9e92-45e8-9764-cbddd88e30d3" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-10-09T05:47:52.908ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-10-09T05:47:52.908ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 199.866883ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5ea1d9b2-f65e-48e4-bb9c-4a5bc60140c3", "content-length": "124", "date": "Wed, 09 Oct 2024 05:47:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5ea1d9b2-f65e-48e4-bb9c-4a5bc60140c3" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-10-09T05:47:53.082ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
1972024-10-09T05:47:53.088ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:33187 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
1982024-10-09T05:47:53.088ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
1992024-10-09T05:47:53.088ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:33187 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 8fa2b36a-fea8-4835-8541-6d298f48930d sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2002024-10-09T05:47:53.089ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 324.151ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2012024-10-09T05:47:53.090ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2022024-10-09T05:47:53.090ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2032024-10-09T05:47:53.090ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:923 underlay_address = fd00:1122:3344:101::1
2042024-10-09T05:47:53.090ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72
2052024-10-09T05:47:53.109ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2062024-10-09T05:47:53.415ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 449.101539ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2072024-10-09T05:47:53.865ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.341529591s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2082024-10-09T05:47:55.208ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.222593467s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2092024-10-09T05:47:55.521ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2699
2102024-10-09T05:47:55.521ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2917
2112024-10-09T05:47:55.521ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2569
2122024-10-09T05:47:55.521ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2616
2132024-10-09T05:47:55.521ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2998
2142024-10-09T05:47:55.521ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3040
2152024-10-09T05:47:55.523ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2842
2162024-10-09T05:47:55.524ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2610
2172024-10-09T05:47:55.525ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='89ba3bee-a184-4fec-8c79-5b1581ccec47'/> <propval type="astring" name="rack_id" value='cde79127-ad1a-4dc8-8bde-c617fda5df2f'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe9e:dade'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='8fa2b36a-fea8-4835-8541-6d298f48930d'/> <propval type="astring" name="rack_id" value='cde79127-ad1a-4dc8-8bde-c617fda5df2f'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='8fa2b36a-fea8-4835-8541-6d298f48930d'/> <propval type="astring" name="rack_uuid" value='cde79127-ad1a-4dc8-8bde-c617fda5df2f'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='8fa2b36a-fea8-4835-8541-6d298f48930d'/> <propval type="astring" name="rack_uuid" value='cde79127-ad1a-4dc8-8bde-c617fda5df2f'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2182024-10-09T05:47:55.525ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2192024-10-09T05:47:55.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 50.461161ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-10-09T05:47:56.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 131.794103ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-10-09T05:47:56.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 264.716374ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-10-09T05:47:56.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 277.793225ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-10-09T05:47:57.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.104892447s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-10-09T05:47:57.433ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.596566694s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2252024-10-09T05:47:58.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.080597458s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-10-09T05:47:59.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 886.362755ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-10-09T05:48:00.032ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 6.740813887s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2282024-10-09T05:48:00.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.456560478s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2292024-10-09T05:48:02.526ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:12345
2302024-10-09T05:48:02.526ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2312024-10-09T05:48:02.526ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2322024-10-09T05:48:02.526ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2332024-10-09T05:48:02.526ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2342024-10-09T05:48:02.526ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2352024-10-09T05:48:02.526ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2362024-10-09T05:48:02.526ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3944
2372024-10-09T05:48:02.734ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2382024-10-09T05:48:02.734ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2392024-10-09T05:48:02.734ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:37556
2402024-10-09T05:48:02.734ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-10-09T05:48:02.734ZINFOSledAgent (RSS): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2422024-10-09T05:48:02.734ZINFOSledAgent (RSS): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2432024-10-09T05:48:02.734ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-10-09T05:48:02.734ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-10-09T05:48:02.734ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-10-09T05:48:02.739ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4265 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2472024-10-09T05:48:02.739ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4281
2482024-10-09T05:48:02.739ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2492024-10-09T05:48:02.984ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52919
2502024-10-09T05:48:02.984ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2512024-10-09T05:48:02.984ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 217 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52919 req_id = 29d6c642-7db5-4d84-82c8-d68f6e079a96 response_code = 200 uri = /inventory
2522024-10-09T05:48:03.014ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64763
2532024-10-09T05:48:03.014ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64763 req_id = dfb3e1a2-b045-4059-8d63-8331f9ecd3b2 response_code = 200 uri = /sled-role
2542024-10-09T05:48:03.014ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2552024-10-09T05:48:03.014ZINFOSledAgent (RSS): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2562024-10-09T05:48:03.015ZINFOSledAgent (RSS): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2572024-10-09T05:48:03.015ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:885
2582024-10-09T05:48:03.046ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 45d0abcb-690e-47b5-94a2-aa2c3beed9c8, pool_id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 52fec25b-86c8-4282-843a-dfd0db0965d2, pool_id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: bfd75524-4021-4869-92d7-e6e88952eb00, pool_id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 8928f720-975d-4e24-90b8-e06817369284, pool_id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 7d0b1e00-86ec-411a-b29f-5a8036411555, pool_id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: ebdb2d66-c159-43b5-aa3f-02808da9571a, pool_id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 86e39ca5-8e91-4aec-8e26-876240db93ca, pool_id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: eb45e973-c9f0-4427-92c9-d9bb89ade5a0, pool_id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 18fba7fc-715b-42bd-af65-6fd4e87945f2, pool_id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2592024-10-09T05:48:03.046ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57512
2602024-10-09T05:48:03.046ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:864 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2612024-10-09T05:48:03.046ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 45d0abcb-690e-47b5-94a2-aa2c3beed9c8, pool_id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 52fec25b-86c8-4282-843a-dfd0db0965d2, pool_id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: bfd75524-4021-4869-92d7-e6e88952eb00, pool_id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 8928f720-975d-4e24-90b8-e06817369284, pool_id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 7d0b1e00-86ec-411a-b29f-5a8036411555, pool_id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: ebdb2d66-c159-43b5-aa3f-02808da9571a, pool_id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 86e39ca5-8e91-4aec-8e26-876240db93ca, pool_id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: eb45e973-c9f0-4427-92c9-d9bb89ade5a0, pool_id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 18fba7fc-715b-42bd-af65-6fd4e87945f2, pool_id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2622024-10-09T05:48:03.046ZINFOSledAgent (StorageManager): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2632024-10-09T05:48:03.046ZINFOSledAgent (StorageManager): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2642024-10-09T05:48:03.046ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2652024-10-09T05:48:03.046ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2662024-10-09T05:48:03.047ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2672024-10-09T05:48:03.047ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2682024-10-09T05:48:03.047ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2692024-10-09T05:48:03.169ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2702024-10-09T05:48:03.169ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2712024-10-09T05:48:03.176ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2722024-10-09T05:48:03.176ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool)
2732024-10-09T05:48:03.379ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2742024-10-09T05:48:03.473ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }
2752024-10-09T05:48:03.480ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2762024-10-09T05:48:03.480ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2772024-10-09T05:48:03.480ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2782024-10-09T05:48:03.480ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2792024-10-09T05:48:03.480ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2802024-10-09T05:48:03.480ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2812024-10-09T05:48:03.480ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2822024-10-09T05:48:03.514ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2832024-10-09T05:48:03.523ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_fe683b07-b738-4a57-88f5-50899c5ceee2/crypt/zone file = sled-storage/src/dataset.rs:280
2842024-10-09T05:48:03.528ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa2b36a-fea8-4835-8541-6d298f48930d) file = sled-agent/src/probe_manager.rs:230 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
2852024-10-09T05:48:03.692ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_fe683b07-b738-4a57-88f5-50899c5ceee2 file = sled-storage/src/dataset.rs:366
2862024-10-09T05:48:03.700ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2872024-10-09T05:48:03.700ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2882024-10-09T05:48:03.700ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2892024-10-09T05:48:03.700ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2902024-10-09T05:48:03.700ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2912024-10-09T05:48:03.710ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2922024-10-09T05:48:03.710ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool)
2932024-10-09T05:48:03.800ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }
2942024-10-09T05:48:03.807ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2952024-10-09T05:48:03.807ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
2962024-10-09T05:48:03.807ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
2972024-10-09T05:48:03.807ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
2982024-10-09T05:48:03.807ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2992024-10-09T05:48:03.807ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3002024-10-09T05:48:03.807ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3012024-10-09T05:48:03.842ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3022024-10-09T05:48:03.849ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f65f0473-59e4-4c49-bb65-a3d5f7f50633/crypt/zone file = sled-storage/src/dataset.rs:280
3032024-10-09T05:48:04.022ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f65f0473-59e4-4c49-bb65-a3d5f7f50633 file = sled-storage/src/dataset.rs:366
3042024-10-09T05:48:04.030ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3052024-10-09T05:48:04.030ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3062024-10-09T05:48:04.030ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3072024-10-09T05:48:04.030ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3082024-10-09T05:48:04.030ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3092024-10-09T05:48:04.039ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3102024-10-09T05:48:04.039ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool)
3112024-10-09T05:48:04.128ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }
3122024-10-09T05:48:04.135ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3132024-10-09T05:48:04.135ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3142024-10-09T05:48:04.135ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3152024-10-09T05:48:04.135ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3162024-10-09T05:48:04.135ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3172024-10-09T05:48:04.136ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3182024-10-09T05:48:04.136ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3192024-10-09T05:48:04.170ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3202024-10-09T05:48:04.177ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f4492cbc-caa2-4fd8-bbf5-d062483d7508/crypt/zone file = sled-storage/src/dataset.rs:280
3212024-10-09T05:48:04.345ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f4492cbc-caa2-4fd8-bbf5-d062483d7508 file = sled-storage/src/dataset.rs:366
3222024-10-09T05:48:04.353ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3232024-10-09T05:48:04.353ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3242024-10-09T05:48:04.353ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3252024-10-09T05:48:04.353ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3262024-10-09T05:48:04.353ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3272024-10-09T05:48:04.364ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3282024-10-09T05:48:04.364ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool)
3292024-10-09T05:48:04.453ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }
3302024-10-09T05:48:04.460ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3312024-10-09T05:48:04.460ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3322024-10-09T05:48:04.460ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3332024-10-09T05:48:04.460ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3342024-10-09T05:48:04.460ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3352024-10-09T05:48:04.461ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3362024-10-09T05:48:04.461ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3372024-10-09T05:48:04.493ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3382024-10-09T05:48:04.501ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6d483f9a-1428-49ad-be75-fa82682b3fb8/crypt/zone file = sled-storage/src/dataset.rs:280
3392024-10-09T05:48:04.530ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa2b36a-fea8-4835-8541-6d298f48930d) file = sled-agent/src/probe_manager.rs:230 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
3402024-10-09T05:48:04.678ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6d483f9a-1428-49ad-be75-fa82682b3fb8 file = sled-storage/src/dataset.rs:366
3412024-10-09T05:48:04.686ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3422024-10-09T05:48:04.686ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3432024-10-09T05:48:04.686ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3442024-10-09T05:48:04.686ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3452024-10-09T05:48:04.686ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3462024-10-09T05:48:04.696ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3472024-10-09T05:48:04.696ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 574723e3-b615-4bd5-bf94-f101903d419b (zpool)
3482024-10-09T05:48:04.787ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }
3492024-10-09T05:48:04.795ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3502024-10-09T05:48:04.795ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3512024-10-09T05:48:04.795ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3522024-10-09T05:48:04.795ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3532024-10-09T05:48:04.795ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3542024-10-09T05:48:04.795ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3552024-10-09T05:48:04.795ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3562024-10-09T05:48:04.830ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3572024-10-09T05:48:04.837ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_574723e3-b615-4bd5-bf94-f101903d419b/crypt/zone file = sled-storage/src/dataset.rs:280
3582024-10-09T05:48:05.008ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_574723e3-b615-4bd5-bf94-f101903d419b file = sled-storage/src/dataset.rs:366
3592024-10-09T05:48:05.018ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3602024-10-09T05:48:05.018ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3612024-10-09T05:48:05.018ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3622024-10-09T05:48:05.018ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3632024-10-09T05:48:05.018ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3642024-10-09T05:48:05.028ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3652024-10-09T05:48:05.028ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool)
3662024-10-09T05:48:05.121ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool), kind: External }
3672024-10-09T05:48:05.129ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3682024-10-09T05:48:05.129ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3692024-10-09T05:48:05.129ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3702024-10-09T05:48:05.129ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3712024-10-09T05:48:05.129ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3722024-10-09T05:48:05.129ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3732024-10-09T05:48:05.129ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3742024-10-09T05:48:05.165ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3752024-10-09T05:48:05.172ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_129ad649-4b7b-42ac-ac7c-60432cf5d7b0/crypt/zone file = sled-storage/src/dataset.rs:280
3762024-10-09T05:48:05.346ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_129ad649-4b7b-42ac-ac7c-60432cf5d7b0 file = sled-storage/src/dataset.rs:366
3772024-10-09T05:48:05.354ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3782024-10-09T05:48:05.354ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3792024-10-09T05:48:05.354ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3802024-10-09T05:48:05.354ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3812024-10-09T05:48:05.354ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3822024-10-09T05:48:05.364ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3832024-10-09T05:48:05.364ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool)
3842024-10-09T05:48:05.456ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }
3852024-10-09T05:48:05.464ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3862024-10-09T05:48:05.465ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3872024-10-09T05:48:05.465ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3882024-10-09T05:48:05.465ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3892024-10-09T05:48:05.465ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3902024-10-09T05:48:05.465ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3912024-10-09T05:48:05.465ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3922024-10-09T05:48:05.500ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3932024-10-09T05:48:05.508ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_55ecf9f3-173f-4dac-bf98-2dda58cc9478/crypt/zone file = sled-storage/src/dataset.rs:280
3942024-10-09T05:48:05.532ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa2b36a-fea8-4835-8541-6d298f48930d) file = sled-agent/src/probe_manager.rs:230 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
3952024-10-09T05:48:05.681ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_55ecf9f3-173f-4dac-bf98-2dda58cc9478 file = sled-storage/src/dataset.rs:366
3962024-10-09T05:48:05.689ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3972024-10-09T05:48:05.689ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3982024-10-09T05:48:05.689ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3992024-10-09T05:48:05.689ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4002024-10-09T05:48:05.689ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4012024-10-09T05:48:05.699ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4022024-10-09T05:48:05.699ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = c4821672-b400-49f1-a3c1-f8485fe155ca (zpool)
4032024-10-09T05:48:05.794ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool), kind: External }
4042024-10-09T05:48:05.802ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4052024-10-09T05:48:05.802ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4062024-10-09T05:48:05.802ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4072024-10-09T05:48:05.802ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4082024-10-09T05:48:05.802ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4092024-10-09T05:48:05.802ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4102024-10-09T05:48:05.802ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4112024-10-09T05:48:05.838ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4122024-10-09T05:48:05.846ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c4821672-b400-49f1-a3c1-f8485fe155ca/crypt/zone file = sled-storage/src/dataset.rs:280
4132024-10-09T05:48:06.034ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c4821672-b400-49f1-a3c1-f8485fe155ca file = sled-storage/src/dataset.rs:366
4142024-10-09T05:48:06.043ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4152024-10-09T05:48:06.043ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4162024-10-09T05:48:06.043ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4172024-10-09T05:48:06.043ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4182024-10-09T05:48:06.043ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4192024-10-09T05:48:06.053ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4202024-10-09T05:48:06.053ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool)
4212024-10-09T05:48:06.152ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool), kind: External }
4222024-10-09T05:48:06.161ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4232024-10-09T05:48:06.161ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4242024-10-09T05:48:06.161ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4252024-10-09T05:48:06.161ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4262024-10-09T05:48:06.161ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4272024-10-09T05:48:06.161ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4282024-10-09T05:48:06.161ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4292024-10-09T05:48:06.197ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4302024-10-09T05:48:06.205ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628/crypt/zone file = sled-storage/src/dataset.rs:280
4312024-10-09T05:48:06.380ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 file = sled-storage/src/dataset.rs:366
4322024-10-09T05:48:06.388ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4332024-10-09T05:48:06.389ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4342024-10-09T05:48:06.389ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4352024-10-09T05:48:06.389ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d6cde337-7634-4a9e-a6ab-bc764b121353 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4362024-10-09T05:48:06.389ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4372024-10-09T05:48:06.389ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4382024-10-09T05:48:06.389ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4392024-10-09T05:48:06.389ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:868 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4402024-10-09T05:48:06.389ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4412024-10-09T05:48:06.389ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:894 generation = Generation(1) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4422024-10-09T05:48:06.389ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:899 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4432024-10-09T05:48:06.389ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:904 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4442024-10-09T05:48:06.389ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4452024-10-09T05:48:06.389ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:910 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4462024-10-09T05:48:06.389ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:798 gen = Generation(1)
4472024-10-09T05:48:06.389ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:915 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4482024-10-09T05:48:06.389ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3342947 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:57512 req_id = 0ce237ba-a13d-4697-aee9-850bf2171c3f response_code = 200 uri = /omicron-physical-disks
4492024-10-09T05:48:06.420ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f51fd25f-beb4-4c48-a50f-4e7c293411bd (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 809d074f-9d26-42fc-9907-735d970f36f3 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4502024-10-09T05:48:06.420ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54748
4512024-10-09T05:48:06.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4522024-10-09T05:48:06.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4532024-10-09T05:48:06.472ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4542024-10-09T05:48:06.472ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4552024-10-09T05:48:06.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4562024-10-09T05:48:06.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4572024-10-09T05:48:06.535ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa2b36a-fea8-4835-8541-6d298f48930d) file = sled-agent/src/probe_manager.rs:230 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4582024-10-09T05:48:06.568ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4592024-10-09T05:48:06.568ZINFOSledAgent (ServiceManager): No ledger in /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-zones.json file = common/src/ledger.rs:188
4602024-10-09T05:48:06.568ZINFOSledAgent (ServiceManager): No ledger in /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-zones.json file = common/src/ledger.rs:188
4612024-10-09T05:48:06.568ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4622024-10-09T05:48:06.609ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4632024-10-09T05:48:06.610ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4642024-10-09T05:48:06.610ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4652024-10-09T05:48:06.661ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd file = illumos-utils/src/zone.rs:316
4662024-10-09T05:48:06.665ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d file = illumos-utils/src/zone.rs:316
4672024-10-09T05:48:06.669ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3 file = illumos-utils/src/zone.rs:316
4682024-10-09T05:48:06.680ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd file = illumos-utils/src/zone.rs:354
4692024-10-09T05:48:06.684ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d file = illumos-utils/src/zone.rs:354
4702024-10-09T05:48:06.688ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3 file = illumos-utils/src/zone.rs:354
4712024-10-09T05:48:06.775ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.701060133s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4722024-10-09T05:48:07.536ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa2b36a-fea8-4835-8541-6d298f48930d) file = sled-agent/src/probe_manager.rs:230 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
4732024-10-09T05:48:08.212ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4742024-10-09T05:48:08.212ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4752024-10-09T05:48:08.690ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4762024-10-09T05:48:08.691ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
4772024-10-09T05:48:09.174ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4782024-10-09T05:48:09.174ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4792024-10-09T05:48:09.174ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4802024-10-09T05:48:09.174ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
4812024-10-09T05:48:09.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 55.270723ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
4822024-10-09T05:48:09.187ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
4832024-10-09T05:48:09.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 131.423931ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
4842024-10-09T05:48:09.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 72.172814ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
4852024-10-09T05:48:09.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 46.746941ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
4862024-10-09T05:48:09.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 76.780926ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
4872024-10-09T05:48:09.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 114.648422ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
4882024-10-09T05:48:09.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 108.551025ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
4892024-10-09T05:48:09.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 222.501014ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
4902024-10-09T05:48:09.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 145.862978ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
4912024-10-09T05:48:09.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 307.21476ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
4922024-10-09T05:48:09.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 520.737265ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
4932024-10-09T05:48:09.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 424.26017ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
4942024-10-09T05:48:09.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 558.907428ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
4952024-10-09T05:48:10.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 879.036677ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
4962024-10-09T05:48:10.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 830.872078ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
4972024-10-09T05:48:10.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 1.080613723s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
4982024-10-09T05:48:11.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 877.433339ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
4992024-10-09T05:48:11.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 1.18391023s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
5002024-10-09T05:48:11.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 543.062802ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
5012024-10-09T05:48:11.846ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4289
5022024-10-09T05:48:11.846ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4299 gateway = fd00:1122:3344:101::1
5032024-10-09T05:48:11.852ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4378
5042024-10-09T05:48:11.961ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4426
5052024-10-09T05:48:11.961ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4453
5062024-10-09T05:48:12.006ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4466
5072024-10-09T05:48:12.006ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4330
5082024-10-09T05:48:12.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 1.447774584s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
5092024-10-09T05:48:12.064ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4372
5102024-10-09T05:48:12.064ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4433 rack_subnet = fd00:1122:3344::
5112024-10-09T05:48:12.085ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4444
5122024-10-09T05:48:12.085ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4486
5132024-10-09T05:48:12.109ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:54338
5142024-10-09T05:48:12.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:54338 req_id = 1ed59570-7c3e-4c8b-8d8d-16b78a1e2527 response_code = 200 uri = /sled-identifiers
5152024-10-09T05:48:12.154ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4518
5162024-10-09T05:48:12.154ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4524
5172024-10-09T05:48:12.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 810.690768ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
5182024-10-09T05:48:12.242ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4565
5192024-10-09T05:48:12.242ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
5202024-10-09T05:48:12.275ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:418
5212024-10-09T05:48:12.306ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:686
5222024-10-09T05:48:12.306ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:724
5232024-10-09T05:48:12.307ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:727 uptime = 0
5242024-10-09T05:48:12.307ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:440
5252024-10-09T05:48:12.343ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
5262024-10-09T05:48:12.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 1.349909456s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
5272024-10-09T05:48:12.390ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5282024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5292024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5302024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5312024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5322024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5332024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5342024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5352024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5362024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5372024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5382024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5392024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5402024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5412024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5422024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5432024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5442024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5452024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5462024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5472024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5482024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5492024-10-09T05:48:12.390ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5502024-10-09T05:48:12.528ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11077829717516143671 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
5512024-10-09T05:48:12.529ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9104562540319307423 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
5522024-10-09T05:48:13.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d") failed: Property not found. retry in 827.390594ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
5532024-10-09T05:48:13.171ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
5542024-10-09T05:48:13.171ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
5552024-10-09T05:48:13.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3") failed: Property not found. retry in 1.068279631s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
5562024-10-09T05:48:13.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd") failed: Property not found. retry in 884.976173ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
5572024-10-09T05:48:14.627ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_internal_dns_f51fd25f-beb4-4c48-a50f-4e7c293411bd
5582024-10-09T05:48:14.627ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_internal_dns_0283ef9e-64d0-4d4f-b0d9-0725e940db9d
5592024-10-09T05:48:14.627ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_internal_dns_809d074f-9d26-42fc-9907-735d970f36f3
5602024-10-09T05:48:14.627ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5612024-10-09T05:48:14.627ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5622024-10-09T05:48:14.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8206901 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:54748 req_id = 44d6e931-ea58-4f8c-8c0c-90aa00bd7131 response_code = 204 uri = /omicron-zones
5632024-10-09T05:48:14.629ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5642024-10-09T05:48:14.676ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5652024-10-09T05:48:14.731ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5662024-10-09T05:48:14.760ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 143.394131ms) dns_config_addr = [fd00:1122:3344:3::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:3::1]:5353/config) file = sled-agent/src/rack_setup/service.rs:595
5672024-10-09T05:48:14.922ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5682024-10-09T05:48:14.922ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5692024-10-09T05:48:14.957ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5702024-10-09T05:48:14.959ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5712024-10-09T05:48:14.959ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5722024-10-09T05:48:14.989ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f51fd25f-beb4-4c48-a50f-4e7c293411bd (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 809d074f-9d26-42fc-9907-735d970f36f3 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 8a22cf94-72e8-4231-80f4-f3e663a70b98 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 677c784d-5cdd-4b47-bd0c-ae49d3e8d286, kind: Service { id: 8a22cf94-72e8-4231-80f4-f3e663a70b98 }, name: Name("ntp-8a22cf94-72e8-4231-80f4-f3e663a70b98"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 252])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5732024-10-09T05:48:14.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61170
5742024-10-09T05:48:14.989ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5752024-10-09T05:48:14.989ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5762024-10-09T05:48:15.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5772024-10-09T05:48:15.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5782024-10-09T05:48:15.027ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5792024-10-09T05:48:15.027ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5802024-10-09T05:48:15.044ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5812024-10-09T05:48:15.044ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-zones.json file = common/src/ledger.rs:177
5822024-10-09T05:48:15.044ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-zones.json file = common/src/ledger.rs:177
5832024-10-09T05:48:15.044ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5842024-10-09T05:48:15.055ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5852024-10-09T05:48:15.088ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5862024-10-09T05:48:15.088ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5872024-10-09T05:48:15.088ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5882024-10-09T05:48:15.118ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 164, 252]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5892024-10-09T05:48:15.118ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = boundary_ntp
5902024-10-09T05:48:15.118ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5912024-10-09T05:48:15.118ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5922024-10-09T05:48:15.118ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5932024-10-09T05:48:15.119ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5942024-10-09T05:48:15.143ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98 file = illumos-utils/src/zone.rs:316
5952024-10-09T05:48:15.162ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98 file = illumos-utils/src/zone.rs:354
5962024-10-09T05:48:15.785ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5972024-10-09T05:48:15.785ZINFOSledAgent (ServiceManager): Booting oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
5982024-10-09T05:48:15.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 39.109193ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
5992024-10-09T05:48:15.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 52.248223ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6002024-10-09T05:48:16.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 237.381412ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6012024-10-09T05:48:16.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 593.40562ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6022024-10-09T05:48:16.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 919.27115ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6032024-10-09T05:48:17.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 1.198501613s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6042024-10-09T05:48:19.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 979.168715ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6052024-10-09T05:48:20.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98") failed: Property not found. retry in 1.043311701s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6062024-10-09T05:48:21.070ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98
6072024-10-09T05:48:21.071ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6082024-10-09T05:48:21.071ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6092024-10-09T05:48:21.071ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6082023 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:61170 req_id = 1aacb526-5ade-40d3-8273-83038d4e62fa response_code = 204 uri = /omicron-zones
6102024-10-09T05:48:21.074ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6112024-10-09T05:48:21.104ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6122024-10-09T05:48:21.104ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46305
6132024-10-09T05:48:21.114ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3897
6142024-10-09T05:48:21.114ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 9298 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46305 req_id = 379a5799-c297-479b-b829-7c2f9338e885 response_code = 500 uri = /timesync
6152024-10-09T05:48:21.117ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6162024-10-09T05:48:21.294ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6172024-10-09T05:48:21.294ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44653
6182024-10-09T05:48:22.531ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11077829717516143671 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6192024-10-09T05:48:22.531ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9104562540319307423 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6202024-10-09T05:48:23.173ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6212024-10-09T05:48:23.173ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6222024-10-09T05:48:23.882ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7493549199027410003 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6232024-10-09T05:48:24.583ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11859151737803065117 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6242024-10-09T05:48:24.629ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12851276588111666858 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6252024-10-09T05:48:25.303ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3897
6262024-10-09T05:48:25.303ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4008934 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44653 req_id = 9b816db4-d125-46e3-a236-65c7c4e68a05 response_code = 500 uri = /timesync
6272024-10-09T05:48:25.306ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6282024-10-09T05:48:25.474ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6292024-10-09T05:48:25.474ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63914
6302024-10-09T05:48:25.482ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3897
6312024-10-09T05:48:25.483ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8597 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63914 req_id = b5f94ba8-d5ff-4eb3-b869-d0daeca4766b response_code = 500 uri = /timesync
6322024-10-09T05:48:25.484ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6332024-10-09T05:48:25.719ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6342024-10-09T05:48:25.719ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40767
6352024-10-09T05:48:25.726ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_8a22cf94-72e8-4231-80f4-f3e663a70b98': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3897
6362024-10-09T05:48:25.726ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 7135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40767 req_id = 5cb806ce-e6da-4105-aa15-9318f62ff3b6 response_code = 500 uri = /timesync
6372024-10-09T05:48:25.728ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6382024-10-09T05:48:25.917ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6392024-10-09T05:48:25.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50336
6402024-10-09T05:48:25.925ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7239 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50336 req_id = fac3ac9c-a378-4ca7-9645-e0f3eff08e69 response_code = 200 uri = /timesync
6412024-10-09T05:48:25.927ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6422024-10-09T05:48:25.927ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6432024-10-09T05:48:26.256ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6442024-10-09T05:48:26.256ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53660
6452024-10-09T05:48:26.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53660 req_id = 7c95c4d1-a344-466c-80a8-de884920ffdd response_code = 200 uri = /timesync
6462024-10-09T05:48:26.266ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6472024-10-09T05:48:26.266ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6482024-10-09T05:48:26.656ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6492024-10-09T05:48:26.656ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49314
6502024-10-09T05:48:26.663ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7386 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49314 req_id = ce119205-11fe-4dfa-b897-41de9e1312de response_code = 200 uri = /timesync
6512024-10-09T05:48:26.665ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6522024-10-09T05:48:26.665ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6532024-10-09T05:48:26.981ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6542024-10-09T05:48:26.982ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37351
6552024-10-09T05:48:26.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7718 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37351 req_id = 7f882606-cb5b-4118-a329-25058726320d response_code = 200 uri = /timesync
6562024-10-09T05:48:26.991ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6572024-10-09T05:48:26.991ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6582024-10-09T05:48:27.260ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6592024-10-09T05:48:27.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52184
6602024-10-09T05:48:27.268ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7806 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52184 req_id = 1e7b9e24-f71c-4819-b107-3fe95d5c080d response_code = 200 uri = /timesync
6612024-10-09T05:48:27.270ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6622024-10-09T05:48:27.270ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6632024-10-09T05:48:28.194ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6642024-10-09T05:48:28.195ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36588
6652024-10-09T05:48:28.202ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7385 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36588 req_id = 44de2d54-994a-499b-a862-ea19ab471d4f response_code = 200 uri = /timesync
6662024-10-09T05:48:28.204ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6672024-10-09T05:48:28.204ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6682024-10-09T05:48:28.740ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6692024-10-09T05:48:28.740ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43129
6702024-10-09T05:48:28.749ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8837 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43129 req_id = 9c75f10a-6408-42c2-8221-8edd219591a3 response_code = 200 uri = /timesync
6712024-10-09T05:48:28.751ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6722024-10-09T05:48:28.751ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6732024-10-09T05:48:29.228ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6742024-10-09T05:48:29.229ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63196
6752024-10-09T05:48:29.237ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8097 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63196 req_id = f008d925-c2f2-41f4-a839-9d238f54e7cb response_code = 200 uri = /timesync
6762024-10-09T05:48:29.238ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6772024-10-09T05:48:29.238ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6782024-10-09T05:48:29.642ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6792024-10-09T05:48:29.642ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48809
6802024-10-09T05:48:29.650ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7942 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48809 req_id = f3845ba3-63cc-4fd8-bd12-c25ef0acac4a response_code = 200 uri = /timesync
6812024-10-09T05:48:29.652ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6822024-10-09T05:48:29.652ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6832024-10-09T05:48:31.020ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6842024-10-09T05:48:31.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63058
6852024-10-09T05:48:31.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8046 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63058 req_id = 605c8929-826c-4301-a0fc-9fbb95420b56 response_code = 200 uri = /timesync
6862024-10-09T05:48:31.030ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: -0.20990786 } file = sled-agent/src/rack_setup/service.rs:660
6872024-10-09T05:48:31.030ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6882024-10-09T05:48:31.471ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9882646737519600752 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6892024-10-09T05:48:31.471ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5476244007219141130 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
6902024-10-09T05:48:32.059ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6912024-10-09T05:48:32.059ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55119
6922024-10-09T05:48:32.068ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8503 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55119 req_id = 4d8a6d53-e953-454d-b29b-4b46427cc0bc response_code = 200 uri = /timesync
6932024-10-09T05:48:32.069ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: -0.204822168 } file = sled-agent/src/rack_setup/service.rs:660
6942024-10-09T05:48:32.069ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6952024-10-09T05:48:32.842ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6962024-10-09T05:48:32.842ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61366
6972024-10-09T05:48:32.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8177 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61366 req_id = 59805ca9-971a-40cf-8ada-20744ad46578 response_code = 200 uri = /timesync
6982024-10-09T05:48:32.854ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: -0.20269607 } file = sled-agent/src/rack_setup/service.rs:660
6992024-10-09T05:48:32.854ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7002024-10-09T05:48:32.907ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 138.416474ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7012024-10-09T05:48:32.926ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11077829717516143671 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7022024-10-09T05:48:32.927ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9104562540319307423 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7032024-10-09T05:48:33.566ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7042024-10-09T05:48:33.566ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7052024-10-09T05:48:34.273ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7493549199027410003 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7062024-10-09T05:48:34.708ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7072024-10-09T05:48:34.708ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56303
7082024-10-09T05:48:34.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8264 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56303 req_id = 966679d5-25cd-4a5b-9533-b841914da1f4 response_code = 200 uri = /timesync
7092024-10-09T05:48:34.718ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452913.710408, correction: -0.191184565 } file = sled-agent/src/rack_setup/service.rs:660
7102024-10-09T05:48:34.718ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7112024-10-09T05:48:34.973ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11859151737803065117 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7122024-10-09T05:48:35.018ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12851276588111666858 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7132024-10-09T05:48:35.881ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7142024-10-09T05:48:35.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36250
7152024-10-09T05:48:35.890ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8031 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36250 req_id = edbed0da-d57f-43c8-8a99-fc0226a48c91 response_code = 200 uri = /timesync
7162024-10-09T05:48:35.891ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452914.750007, correction: -0.187506273 } file = sled-agent/src/rack_setup/service.rs:660
7172024-10-09T05:48:35.891ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7182024-10-09T05:48:38.176ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7192024-10-09T05:48:38.176ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42810
7202024-10-09T05:48:38.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9346 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42810 req_id = 0506e50f-b3ae-436a-be93-45e48e7d6ed0 response_code = 200 uri = /timesync
7212024-10-09T05:48:38.187ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452917.1271603, correction: -0.179696172 } file = sled-agent/src/rack_setup/service.rs:660
7222024-10-09T05:48:38.187ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7232024-10-09T05:48:39.932ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7242024-10-09T05:48:39.932ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43216
7252024-10-09T05:48:39.941ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8683 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43216 req_id = df927ef1-171d-45a7-bdb6-3c441ed463b6 response_code = 200 uri = /timesync
7262024-10-09T05:48:39.942ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452918.222392, correction: -0.174570709 } file = sled-agent/src/rack_setup/service.rs:660
7272024-10-09T05:48:39.942ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7282024-10-09T05:48:41.444ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9882646737519600752 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7292024-10-09T05:48:41.444ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5476244007219141130 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7302024-10-09T05:48:41.569ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7312024-10-09T05:48:41.569ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59827
7322024-10-09T05:48:41.577ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8241 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59827 req_id = a228fb71-b0de-43f5-bd55-67d0b386f9ff response_code = 200 uri = /timesync
7332024-10-09T05:48:41.580ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452920.616716, correction: -0.169761896 } file = sled-agent/src/rack_setup/service.rs:660
7342024-10-09T05:48:41.580ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7352024-10-09T05:48:42.418ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33713
7362024-10-09T05:48:42.418ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 64 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33713 req_id = f813a30c-481e-4a31-a308-d0b4a38bb580 response_code = 200 uri = /baseboard
7372024-10-09T05:48:42.901ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11077829717516143671 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7382024-10-09T05:48:42.901ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9104562540319307423 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7392024-10-09T05:48:43.539ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7402024-10-09T05:48:43.540ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7412024-10-09T05:48:43.656ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7422024-10-09T05:48:43.656ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46442
7432024-10-09T05:48:43.664ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8337 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46442 req_id = fea00b82-8e36-4cda-a1ca-2058899bb73b response_code = 200 uri = /timesync
7442024-10-09T05:48:43.667ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452922.5857987, correction: -0.164280504 } file = sled-agent/src/rack_setup/service.rs:660
7452024-10-09T05:48:43.667ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7462024-10-09T05:48:44.246ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7493549199027410003 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7472024-10-09T05:48:44.946ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11859151737803065117 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7482024-10-09T05:48:44.990ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12851276588111666858 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7492024-10-09T05:48:46.148ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7502024-10-09T05:48:46.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48425
7512024-10-09T05:48:46.157ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8290 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48425 req_id = 083fcc7d-76d4-44c2-9978-8ce5bec305f3 response_code = 200 uri = /timesync
7522024-10-09T05:48:46.159ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452925.9541574, correction: -0.157658532 } file = sled-agent/src/rack_setup/service.rs:660
7532024-10-09T05:48:46.159ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7542024-10-09T05:48:48.039ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 262.423093ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7552024-10-09T05:48:48.987ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7562024-10-09T05:48:48.988ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42398
7572024-10-09T05:48:48.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8520 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42398 req_id = 759dbfb3-20d3-4ae9-983d-be412aaa99ba response_code = 200 uri = /timesync
7582024-10-09T05:48:48.999ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452927.4147267, correction: -0.150031075 } file = sled-agent/src/rack_setup/service.rs:660
7592024-10-09T05:48:48.999ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7602024-10-09T05:48:51.419ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9882646737519600752 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7612024-10-09T05:48:51.419ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5476244007219141130 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7622024-10-09T05:48:52.876ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11077829717516143671 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7632024-10-09T05:48:52.876ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9104562540319307423 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7642024-10-09T05:48:53.514ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7652024-10-09T05:48:53.514ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7662024-10-09T05:48:53.728ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7672024-10-09T05:48:53.728ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49842
7682024-10-09T05:48:53.737ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8863 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49842 req_id = b1053708-f89f-4016-8a78-eef58f1e2ca8 response_code = 200 uri = /timesync
7692024-10-09T05:48:53.740ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452932.9539552, correction: -0.137153685 } file = sled-agent/src/rack_setup/service.rs:660
7702024-10-09T05:48:53.740ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7712024-10-09T05:48:54.222ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7493549199027410003 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7722024-10-09T05:48:54.920ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11859151737803065117 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7732024-10-09T05:48:54.964ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12851276588111666858 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7742024-10-09T05:48:59.564ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7752024-10-09T05:48:59.565ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54015
7762024-10-09T05:48:59.573ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8414 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54015 req_id = 707c4099-f582-4c34-9217-dd6726f55be0 response_code = 200 uri = /timesync
7772024-10-09T05:48:59.575ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452937.3173862, correction: -0.121294446 } file = sled-agent/src/rack_setup/service.rs:660
7782024-10-09T05:48:59.575ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7792024-10-09T05:49:01.394ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9882646737519600752 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7802024-10-09T05:49:01.394ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5476244007219141130 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7812024-10-09T05:49:02.852ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 11077829717516143671 reason = Duration(60s) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7822024-10-09T05:49:02.853ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 9104562540319307423 reason = Duration(60s) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7832024-10-09T05:49:03.295ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 841.081623ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7842024-10-09T05:49:03.489ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7852024-10-09T05:49:03.489ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7862024-10-09T05:49:04.196ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7493549199027410003 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7872024-10-09T05:49:04.894ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11859151737803065117 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7882024-10-09T05:49:04.939ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12851276588111666858 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7892024-10-09T05:49:08.164ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7902024-10-09T05:49:08.164ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61270
7912024-10-09T05:49:08.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8976 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61270 req_id = c41b9703-7847-480a-9e61-67b08e6498d0 response_code = 200 uri = /timesync
7922024-10-09T05:49:08.175ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452943.32583, correction: -0.097975008 } file = sled-agent/src/rack_setup/service.rs:660
7932024-10-09T05:49:08.175ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7942024-10-09T05:49:11.370ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9882646737519600752 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7952024-10-09T05:49:11.370ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5476244007219141130 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7962024-10-09T05:49:12.376ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58724
7972024-10-09T05:49:12.376ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 36 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58724 req_id = 184807c8-a1be-41e7-b968-913de52bdc2a response_code = 200 uri = /baseboard
7982024-10-09T05:49:13.465ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
7992024-10-09T05:49:13.465ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8002024-10-09T05:49:14.171ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 7493549199027410003 reason = Duration(60s) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8012024-10-09T05:49:14.868ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 11859151737803065117 reason = Duration(60s) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8022024-10-09T05:49:14.913ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 12851276588111666858 reason = Duration(60s) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8032024-10-09T05:49:19.128ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.288552186s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8042024-10-09T05:49:21.344ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 9882646737519600752 reason = Duration(60s) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8052024-10-09T05:49:21.344ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 5476244007219141130 reason = Duration(60s) sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8062024-10-09T05:49:23.439ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8072024-10-09T05:49:23.439ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8082024-10-09T05:49:24.869ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8092024-10-09T05:49:24.870ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56866
8102024-10-09T05:49:24.878ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8367 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56866 req_id = 93e0adda-eb79-42d8-b836-29a135834035 response_code = 200 uri = /timesync
8112024-10-09T05:49:24.881ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452959.8650494, correction: -0.052617874 } file = sled-agent/src/rack_setup/service.rs:660
8122024-10-09T05:49:24.881ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8132024-10-09T05:49:33.414ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15846663927691493982 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8142024-10-09T05:49:33.414ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 17121165301499759131 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8152024-10-09T05:49:36.404ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.151675653s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8162024-10-09T05:49:38.898ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8172024-10-09T05:49:38.898ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45214
8182024-10-09T05:49:38.918ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 19451 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45214 req_id = 3bb5187d-3f95-4899-9827-961427c2a57d response_code = 200 uri = /timesync
8192024-10-09T05:49:38.920ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-09T05:49:02.852353591Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 11077829717516143671 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8202024-10-09T05:49:38.920ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-09T05:49:14.913475984Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 12851276588111666858 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8212024-10-09T05:49:38.920ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-09T05:49:14.171492526Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 7493549199027410003 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8222024-10-09T05:49:38.920ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-09T05:49:21.344431465Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 9882646737519600752 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8232024-10-09T05:49:38.921ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-09T05:49:02.853103004Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 9104562540319307423 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8242024-10-09T05:49:38.921ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-09T05:49:14.868188348Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 11859151737803065117 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8252024-10-09T05:49:38.921ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-09T05:49:21.344683279Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 5476244007219141130 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
8262024-10-09T05:49:38.922ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 1755830211, ip_addr: 104.167.215.195, stratum: 3, ref_time: 1728452975.8919013, correction: -0.014514958 } file = sled-agent/src/rack_setup/service.rs:660
8272024-10-09T05:49:38.922ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1299
8282024-10-09T05:49:38.951ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f51fd25f-beb4-4c48-a50f-4e7c293411bd (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 809d074f-9d26-42fc-9907-735d970f36f3 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 8a22cf94-72e8-4231-80f4-f3e663a70b98 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 677c784d-5cdd-4b47-bd0c-ae49d3e8d286, kind: Service { id: 8a22cf94-72e8-4231-80f4-f3e663a70b98 }, name: Name("ntp-8a22cf94-72e8-4231-80f4-f3e663a70b98"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 252])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7d8092fc-765a-4694-a38b-bd2c03840c4b (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f2ad625-5d08-4939-bbdb-25251decfddb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } } } }, OmicronZoneConfig { id: 98c12b3e-7055-4db5-aa92-ad85b3cf7fdd (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External } } } }, OmicronZoneConfig { id: e04e97c1-38d9-43a9-b6ef-80a4426a76cc (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8292024-10-09T05:49:38.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48378
8302024-10-09T05:49:38.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8312024-10-09T05:49:38.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8322024-10-09T05:49:38.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8332024-10-09T05:49:38.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8342024-10-09T05:49:38.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8352024-10-09T05:49:38.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8362024-10-09T05:49:39.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8372024-10-09T05:49:39.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8382024-10-09T05:49:39.061ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7d8092fc-765a-4694-a38b-bd2c03840c4b, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8392024-10-09T05:49:39.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7d8092fc-765a-4694-a38b-bd2c03840c4b, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8402024-10-09T05:49:39.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f2ad625-5d08-4939-bbdb-25251decfddb, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8412024-10-09T05:49:39.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3f2ad625-5d08-4939-bbdb-25251decfddb, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8422024-10-09T05:49:39.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98c12b3e-7055-4db5-aa92-ad85b3cf7fdd, dataset_name: DatasetName { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8432024-10-09T05:49:39.164ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 98c12b3e-7055-4db5-aa92-ad85b3cf7fdd, dataset_name: DatasetName { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8442024-10-09T05:49:39.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e04e97c1-38d9-43a9-b6ef-80a4426a76cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8452024-10-09T05:49:39.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e04e97c1-38d9-43a9-b6ef-80a4426a76cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }, kind: Cockroach }, responder: ... }
8462024-10-09T05:49:39.267ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8472024-10-09T05:49:39.268ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-zones.json file = common/src/ledger.rs:177
8482024-10-09T05:49:39.268ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-zones.json file = common/src/ledger.rs:177
8492024-10-09T05:49:39.268ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8502024-10-09T05:49:39.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8512024-10-09T05:49:39.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8522024-10-09T05:49:39.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8532024-10-09T05:49:39.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8542024-10-09T05:49:39.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8552024-10-09T05:49:39.438ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd file = illumos-utils/src/zone.rs:316
8562024-10-09T05:49:39.442ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b file = illumos-utils/src/zone.rs:316
8572024-10-09T05:49:39.446ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb file = illumos-utils/src/zone.rs:316
8582024-10-09T05:49:39.450ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0 file = illumos-utils/src/zone.rs:316
8592024-10-09T05:49:39.454ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc file = illumos-utils/src/zone.rs:316
8602024-10-09T05:49:39.458ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd file = illumos-utils/src/zone.rs:354
8612024-10-09T05:49:39.462ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b file = illumos-utils/src/zone.rs:354
8622024-10-09T05:49:39.466ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb file = illumos-utils/src/zone.rs:354
8632024-10-09T05:49:39.471ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0 file = illumos-utils/src/zone.rs:354
8642024-10-09T05:49:39.475ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc file = illumos-utils/src/zone.rs:354
8652024-10-09T05:49:42.328ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65334
8662024-10-09T05:49:42.328ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 58 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65334 req_id = 96a1f884-bed1-42c0-867f-f0cc9faeac67 response_code = 200 uri = /baseboard
8672024-10-09T05:49:42.376ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3f2ad625-5d08-4939-bbdb-25251decfddb'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8682024-10-09T05:49:42.376ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
8692024-10-09T05:49:42.383ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='98c12b3e-7055-4db5-aa92-ad85b3cf7fdd'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8702024-10-09T05:49:42.383ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
8712024-10-09T05:49:42.489ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='7d8092fc-765a-4694-a38b-bd2c03840c4b'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8722024-10-09T05:49:42.489ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
8732024-10-09T05:49:42.495ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e04e97c1-38d9-43a9-b6ef-80a4426a76cc'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8742024-10-09T05:49:42.495ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
8752024-10-09T05:49:42.500ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8762024-10-09T05:49:42.500ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
8772024-10-09T05:49:42.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 32.094323ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
8782024-10-09T05:49:42.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 48.404882ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
8792024-10-09T05:49:42.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 129.940467ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
8802024-10-09T05:49:42.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 54.896353ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
8812024-10-09T05:49:42.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 283.491832ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
8822024-10-09T05:49:42.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 196.07801ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
8832024-10-09T05:49:42.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 28.120368ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
8842024-10-09T05:49:42.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 33.291895ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
8852024-10-09T05:49:42.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 63.421997ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
8862024-10-09T05:49:42.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 61.80621ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
8872024-10-09T05:49:42.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 66.310584ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
8882024-10-09T05:49:43.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 297.760816ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
8892024-10-09T05:49:43.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 136.254776ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
8902024-10-09T05:49:43.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 590.583067ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
8912024-10-09T05:49:43.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 85.629666ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
8922024-10-09T05:49:43.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 138.763276ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
8932024-10-09T05:49:43.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 1.076708411s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
8942024-10-09T05:49:43.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 295.33542ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
8952024-10-09T05:49:43.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 116.2381ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
8962024-10-09T05:49:43.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 463.122464ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
8972024-10-09T05:49:43.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 289.582726ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
8982024-10-09T05:49:43.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.130614147s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
8992024-10-09T05:49:44.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 603.696338ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9002024-10-09T05:49:44.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 783.594662ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9012024-10-09T05:49:44.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 647.254029ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9022024-10-09T05:49:44.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 952.75575ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9032024-10-09T05:49:44.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 967.301064ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9042024-10-09T05:49:44.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 1.39098242s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9052024-10-09T05:49:45.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 565.413578ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9062024-10-09T05:49:45.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 597.962721ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9072024-10-09T05:49:45.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 1.229779867s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9082024-10-09T05:49:45.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 1.293487383s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9092024-10-09T05:49:45.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 631.335273ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9102024-10-09T05:49:45.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.319492896s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9112024-10-09T05:49:46.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 1.423065915s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9122024-10-09T05:49:46.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 949.038095ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9132024-10-09T05:49:46.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 713.715317ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9142024-10-09T05:49:46.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 971.145811ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9152024-10-09T05:49:47.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 571.967229ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9162024-10-09T05:49:47.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 966.435121ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9172024-10-09T05:49:47.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 832.425722ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9182024-10-09T05:49:47.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 547.929785ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9192024-10-09T05:49:47.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 806.238793ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9202024-10-09T05:49:47.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 634.557489ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9212024-10-09T05:49:48.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.222525243s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9222024-10-09T05:49:48.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 1.062055727s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9232024-10-09T05:49:48.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 1.097175509s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9242024-10-09T05:49:48.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 541.776054ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9252024-10-09T05:49:48.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 1.21061506s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9262024-10-09T05:49:49.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 659.788472ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9272024-10-09T05:49:49.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 1.246814414s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9282024-10-09T05:49:49.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 925.842709ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9292024-10-09T05:49:49.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0") failed: Property not found. retry in 1.130213693s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9302024-10-09T05:49:49.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 589.185624ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9312024-10-09T05:49:49.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 557.989959ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9322024-10-09T05:49:50.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 697.147731ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9332024-10-09T05:49:50.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd") failed: Property not found. retry in 1.377786779s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9342024-10-09T05:49:50.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb") failed: Property not found. retry in 875.529287ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9352024-10-09T05:49:50.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc") failed: Property not found. retry in 557.828595ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9362024-10-09T05:49:51.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 811.43514ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9372024-10-09T05:49:51.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 566.414871ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9382024-10-09T05:49:52.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 891.803181ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9392024-10-09T05:49:53.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.203194912s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9402024-10-09T05:49:53.568ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 10.958569316s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
9412024-10-09T05:49:54.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.414553755s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9422024-10-09T05:49:56.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 688.12421ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9432024-10-09T05:49:56.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.497438139s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9442024-10-09T05:49:58.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.20550577s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9452024-10-09T05:49:59.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 1.186213659s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9462024-10-09T05:50:00.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b") failed: Property not found. retry in 999.008326ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9472024-10-09T05:50:01.643ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_cockroachdb_98c12b3e-7055-4db5-aa92-ad85b3cf7fdd
9482024-10-09T05:50:01.643ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_cockroachdb_7d8092fc-765a-4694-a38b-bd2c03840c4b
9492024-10-09T05:50:01.643ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_cockroachdb_3f2ad625-5d08-4939-bbdb-25251decfddb
9502024-10-09T05:50:01.644ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_cockroachdb_e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0
9512024-10-09T05:50:01.644ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_cockroachdb_e04e97c1-38d9-43a9-b6ef-80a4426a76cc
9522024-10-09T05:50:01.644ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9532024-10-09T05:50:01.644ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9542024-10-09T05:50:01.645ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22709468 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48378 req_id = 33805ece-7a8f-4b34-ad33-4610f68a0feb response_code = 204 uri = /omicron-zones
9552024-10-09T05:50:01.681ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40340
9562024-10-09T05:50:01.714ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:3740
9572024-10-09T05:50:04.993ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3758
9582024-10-09T05:50:07.627ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3781
9592024-10-09T05:50:07.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 5945915 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40340 req_id = c36a36e4-3ec9-420f-a955-25bd23711663 response_code = 204 uri = /cockroachdb
9602024-10-09T05:50:07.749ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: f51fd25f-beb4-4c48-a50f-4e7c293411bd (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 809d074f-9d26-42fc-9907-735d970f36f3 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 8a22cf94-72e8-4231-80f4-f3e663a70b98 (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 677c784d-5cdd-4b47-bd0c-ae49d3e8d286, kind: Service { id: 8a22cf94-72e8-4231-80f4-f3e663a70b98 }, name: Name("ntp-8a22cf94-72e8-4231-80f4-f3e663a70b98"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 252])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7d8092fc-765a-4694-a38b-bd2c03840c4b (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f2ad625-5d08-4939-bbdb-25251decfddb (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } } } }, OmicronZoneConfig { id: 98c12b3e-7055-4db5-aa92-ad85b3cf7fdd (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External } } } }, OmicronZoneConfig { id: e04e97c1-38d9-43a9-b6ef-80a4426a76cc (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External } } } }, OmicronZoneConfig { id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 653f6089-0158-484c-84b9-bfd9e4d86b4f, kind: Service { id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 }, name: Name("external-dns-d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 211, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 94415647-751f-4321-816f-7e4752b52496 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: cad5ed0d-3ea9-489d-93d4-7e3565b87eb6, kind: Service { id: 94415647-751f-4321-816f-7e4752b52496 }, name: Name("external-dns-94415647-751f-4321-816f-7e4752b52496"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 7])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: bcd392d4-4bfb-4d15-99aa-1265e84bc8dc (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6296d238-bc00-44df-a0e2-f8ef31c024da, kind: Service { id: bcd392d4-4bfb-4d15-99aa-1265e84bc8dc }, name: Name("nexus-bcd392d4-4bfb-4d15-99aa-1265e84bc8dc"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 191, 204])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 9a7d69b2-b5cb-4929-9d3e-5c8730f41abc (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 7f2385eb-ba1c-4d5f-9e21-61e91356244e, kind: Service { id: 9a7d69b2-b5cb-4929-9d3e-5c8730f41abc }, name: Name("nexus-9a7d69b2-b5cb-4929-9d3e-5c8730f41abc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 96])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: a9346314-f5a8-4da4-8c60-48c5ab6c3dee (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 08db3328-de2f-4bc0-aa29-88948e0b323e, kind: Service { id: a9346314-f5a8-4da4-8c60-48c5ab6c3dee }, name: Name("nexus-a9346314-f5a8-4da4-8c60-48c5ab6c3dee"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8b21fd27-df66-4b03-ac3d-86442491921a (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 9591b96e-20a2-403d-a1c3-ac848f6a8ae1 (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 74b33d8f-4ccf-4b0a-a237-609d67e5f664 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 44f1da2d-5239-4bd4-9324-c1144628f93d (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: f85f5cb6-5308-4cab-9f33-fd8db8cdc51d (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: f7b6d332-9837-40d3-96a7-e562f6f28797 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External } } } }, OmicronZoneConfig { id: bea743e6-cd46-4ddc-84f1-3d8e99a5995a (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External } } } }, OmicronZoneConfig { id: 644e9307-d700-417a-856e-c71809b73b46 (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External } } } }, OmicronZoneConfig { id: 11ca442c-f019-48a8-a842-a0f49e30b7a8 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External } } } }, OmicronZoneConfig { id: f6002c58-4bdb-4ab2-a759-d7920f1c1bc2 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External } } } }, OmicronZoneConfig { id: dcb46d89-108c-4535-bdf4-4263a2f94554 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4db41e00-66ff-487e-8c30-c084b43ec6de (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External } } } }, OmicronZoneConfig { id: f4690efb-90e0-497f-9635-de08dea99416 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool), kind: External } } } }, OmicronZoneConfig { id: a0bf7df0-ebbb-414f-a736-03b7e818e235 (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9612024-10-09T05:50:07.749ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37220
9622024-10-09T05:50:07.750ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9632024-10-09T05:50:07.750ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0283ef9e-64d0-4d4f-b0d9-0725e940db9d, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9642024-10-09T05:50:07.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9652024-10-09T05:50:07.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f51fd25f-beb4-4c48-a50f-4e7c293411bd, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9662024-10-09T05:50:07.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9672024-10-09T05:50:07.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 809d074f-9d26-42fc-9907-735d970f36f3, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9682024-10-09T05:50:07.814ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9692024-10-09T05:50:07.814ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e342e717-aa3e-4ddd-b3f5-d3f9dacc29d0, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9702024-10-09T05:50:07.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7d8092fc-765a-4694-a38b-bd2c03840c4b, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9712024-10-09T05:50:07.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7d8092fc-765a-4694-a38b-bd2c03840c4b, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9722024-10-09T05:50:07.851ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f2ad625-5d08-4939-bbdb-25251decfddb, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9732024-10-09T05:50:07.851ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3f2ad625-5d08-4939-bbdb-25251decfddb, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9742024-10-09T05:50:07.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98c12b3e-7055-4db5-aa92-ad85b3cf7fdd, dataset_name: DatasetName { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9752024-10-09T05:50:07.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 98c12b3e-7055-4db5-aa92-ad85b3cf7fdd, dataset_name: DatasetName { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9762024-10-09T05:50:07.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e04e97c1-38d9-43a9-b6ef-80a4426a76cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9772024-10-09T05:50:07.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e04e97c1-38d9-43a9-b6ef-80a4426a76cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }, kind: Cockroach }, responder: ... }
9782024-10-09T05:50:07.915ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9792024-10-09T05:50:07.915ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9802024-10-09T05:50:07.988ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 94415647-751f-4321-816f-7e4752b52496, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9812024-10-09T05:50:07.988ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 94415647-751f-4321-816f-7e4752b52496, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9822024-10-09T05:50:08.066ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9591b96e-20a2-403d-a1c3-ac848f6a8ae1, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
9832024-10-09T05:50:08.066ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 9591b96e-20a2-403d-a1c3-ac848f6a8ae1, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9842024-10-09T05:50:08.135ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7b6d332-9837-40d3-96a7-e562f6f28797, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9852024-10-09T05:50:08.135ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f7b6d332-9837-40d3-96a7-e562f6f28797, dataset_name: DatasetName { pool_name: ZpoolName { id: fe683b07-b738-4a57-88f5-50899c5ceee2 (zpool), kind: External }, kind: Crucible }, responder: ... }
9862024-10-09T05:50:08.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bea743e6-cd46-4ddc-84f1-3d8e99a5995a, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9872024-10-09T05:50:08.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: bea743e6-cd46-4ddc-84f1-3d8e99a5995a, dataset_name: DatasetName { pool_name: ZpoolName { id: f65f0473-59e4-4c49-bb65-a3d5f7f50633 (zpool), kind: External }, kind: Crucible }, responder: ... }
9882024-10-09T05:50:08.255ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 644e9307-d700-417a-856e-c71809b73b46, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9892024-10-09T05:50:08.255ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 644e9307-d700-417a-856e-c71809b73b46, dataset_name: DatasetName { pool_name: ZpoolName { id: f4492cbc-caa2-4fd8-bbf5-d062483d7508 (zpool), kind: External }, kind: Crucible }, responder: ... }
9902024-10-09T05:50:08.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 11ca442c-f019-48a8-a842-a0f49e30b7a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9912024-10-09T05:50:08.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 11ca442c-f019-48a8-a842-a0f49e30b7a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6d483f9a-1428-49ad-be75-fa82682b3fb8 (zpool), kind: External }, kind: Crucible }, responder: ... }
9922024-10-09T05:50:08.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f6002c58-4bdb-4ab2-a759-d7920f1c1bc2, dataset_name: DatasetName { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9932024-10-09T05:50:08.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f6002c58-4bdb-4ab2-a759-d7920f1c1bc2, dataset_name: DatasetName { pool_name: ZpoolName { id: 574723e3-b615-4bd5-bf94-f101903d419b (zpool), kind: External }, kind: Crucible }, responder: ... }
9942024-10-09T05:50:08.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dcb46d89-108c-4535-bdf4-4263a2f94554, dataset_name: DatasetName { pool_name: ZpoolName { id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9952024-10-09T05:50:08.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dcb46d89-108c-4535-bdf4-4263a2f94554, dataset_name: DatasetName { pool_name: ZpoolName { id: 129ad649-4b7b-42ac-ac7c-60432cf5d7b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
9962024-10-09T05:50:08.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4db41e00-66ff-487e-8c30-c084b43ec6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9972024-10-09T05:50:08.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 4db41e00-66ff-487e-8c30-c084b43ec6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 55ecf9f3-173f-4dac-bf98-2dda58cc9478 (zpool), kind: External }, kind: Crucible }, responder: ... }
9982024-10-09T05:50:08.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f4690efb-90e0-497f-9635-de08dea99416, dataset_name: DatasetName { pool_name: ZpoolName { id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
9992024-10-09T05:50:08.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f4690efb-90e0-497f-9635-de08dea99416, dataset_name: DatasetName { pool_name: ZpoolName { id: c4821672-b400-49f1-a3c1-f8485fe155ca (zpool), kind: External }, kind: Crucible }, responder: ... }
10002024-10-09T05:50:08.657ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0bf7df0-ebbb-414f-a736-03b7e818e235, dataset_name: DatasetName { pool_name: ZpoolName { id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10012024-10-09T05:50:08.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a0bf7df0-ebbb-414f-a736-03b7e818e235, dataset_name: DatasetName { pool_name: ZpoolName { id: 8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628 (zpool), kind: External }, kind: Crucible }, responder: ... }
10022024-10-09T05:50:08.724ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10032024-10-09T05:50:08.724ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-zones.json file = common/src/ledger.rs:177
10042024-10-09T05:50:08.724ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-zones.json file = common/src/ledger.rs:177
10052024-10-09T05:50:08.724ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10062024-10-09T05:50:08.877ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10072024-10-09T05:50:08.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10082024-10-09T05:50:08.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10092024-10-09T05:50:08.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10102024-10-09T05:50:08.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10112024-10-09T05:50:08.879ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10122024-10-09T05:50:08.880ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10132024-10-09T05:50:08.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10142024-10-09T05:50:08.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10152024-10-09T05:50:08.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10162024-10-09T05:50:09.088ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10172024-10-09T05:50:09.088ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10182024-10-09T05:50:09.088ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10192024-10-09T05:50:09.088ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10202024-10-09T05:50:09.116ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10212024-10-09T05:50:09.116ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10222024-10-09T05:50:09.116ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10232024-10-09T05:50:09.116ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10242024-10-09T05:50:09.128ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10252024-10-09T05:50:09.129ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2 file = illumos-utils/src/zone.rs:316
10262024-10-09T05:50:09.134ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554 file = illumos-utils/src/zone.rs:316
10272024-10-09T05:50:09.140ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a file = illumos-utils/src/zone.rs:316
10282024-10-09T05:50:09.145ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_644e9307-d700-417a-856e-c71809b73b46 file = illumos-utils/src/zone.rs:316
10292024-10-09T05:50:09.151ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1 file = illumos-utils/src/zone.rs:316
10302024-10-09T05:50:09.157ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664 file = illumos-utils/src/zone.rs:316
10312024-10-09T05:50:09.162ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d file = illumos-utils/src/zone.rs:316
10322024-10-09T05:50:09.308ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2 file = illumos-utils/src/zone.rs:354
10332024-10-09T05:50:09.312ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554 file = illumos-utils/src/zone.rs:354
10342024-10-09T05:50:09.317ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a file = illumos-utils/src/zone.rs:354
10352024-10-09T05:50:09.321ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_644e9307-d700-417a-856e-c71809b73b46 file = illumos-utils/src/zone.rs:354
10362024-10-09T05:50:09.327ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1 file = illumos-utils/src/zone.rs:354
10372024-10-09T05:50:09.332ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664 file = illumos-utils/src/zone.rs:354
10382024-10-09T05:50:09.337ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d file = illumos-utils/src/zone.rs:354
10392024-10-09T05:50:09.342ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a file = illumos-utils/src/zone.rs:316
10402024-10-09T05:50:09.346ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de file = illumos-utils/src/zone.rs:316
10412024-10-09T05:50:09.352ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416 file = illumos-utils/src/zone.rs:316
10422024-10-09T05:50:09.357ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235 file = illumos-utils/src/zone.rs:316
10432024-10-09T05:50:09.362ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797 file = illumos-utils/src/zone.rs:316
10442024-10-09T05:50:09.368ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d file = illumos-utils/src/zone.rs:316
10452024-10-09T05:50:09.374ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8 file = illumos-utils/src/zone.rs:316
10462024-10-09T05:50:09.427ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10472024-10-09T05:50:09.427ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a file = illumos-utils/src/zone.rs:354
10482024-10-09T05:50:09.434ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de file = illumos-utils/src/zone.rs:354
10492024-10-09T05:50:09.440ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416 file = illumos-utils/src/zone.rs:354
10502024-10-09T05:50:09.447ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235 file = illumos-utils/src/zone.rs:354
10512024-10-09T05:50:09.455ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797 file = illumos-utils/src/zone.rs:354
10522024-10-09T05:50:09.462ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d file = illumos-utils/src/zone.rs:354
10532024-10-09T05:50:09.470ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8 file = illumos-utils/src/zone.rs:354
10542024-10-09T05:50:09.594ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10552024-10-09T05:50:09.722ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10562024-10-09T05:50:09.921ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10572024-10-09T05:50:09.975ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10582024-10-09T05:50:09.978ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10592024-10-09T05:50:09.978ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10602024-10-09T05:50:10.031ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 147, 7]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
10612024-10-09T05:50:10.031ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
10622024-10-09T05:50:10.034ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10632024-10-09T05:50:10.034ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10642024-10-09T05:50:10.087ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 191, 204]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10652024-10-09T05:50:10.087ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
10662024-10-09T05:50:10.087ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
10672024-10-09T05:50:10.087ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
10682024-10-09T05:50:10.089ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10692024-10-09T05:50:10.089ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10702024-10-09T05:50:10.150ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 180, 96]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10712024-10-09T05:50:10.150ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
10722024-10-09T05:50:10.153ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10732024-10-09T05:50:10.153ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10742024-10-09T05:50:10.209ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 134, 29]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10752024-10-09T05:50:10.209ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
10762024-10-09T05:50:10.210ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10772024-10-09T05:50:10.210ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10782024-10-09T05:50:10.268ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 211, 149]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
10792024-10-09T05:50:10.268ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
10802024-10-09T05:50:10.269ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
10812024-10-09T05:50:10.271ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10822024-10-09T05:50:10.347ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
10832024-10-09T05:50:10.347ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
10842024-10-09T05:50:10.347ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
10852024-10-09T05:50:10.347ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
10862024-10-09T05:50:10.348ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
10872024-10-09T05:50:10.348ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
10882024-10-09T05:50:10.348ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
10892024-10-09T05:50:10.348ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
10902024-10-09T05:50:10.348ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
10912024-10-09T05:50:10.351ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
10922024-10-09T05:50:10.357ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10932024-10-09T05:50:10.357ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
10942024-10-09T05:50:10.357ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10952024-10-09T05:50:10.360ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
10962024-10-09T05:50:10.364ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10972024-10-09T05:50:10.364ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10982024-10-09T05:50:10.849ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_94415647-751f-4321-816f-7e4752b52496 file = illumos-utils/src/zone.rs:316
10992024-10-09T05:50:11.058ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 file = illumos-utils/src/zone.rs:316
11002024-10-09T05:50:11.147ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc file = illumos-utils/src/zone.rs:316
11012024-10-09T05:50:11.370ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_94415647-751f-4321-816f-7e4752b52496 file = illumos-utils/src/zone.rs:354
11022024-10-09T05:50:11.385ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc file = illumos-utils/src/zone.rs:316
11032024-10-09T05:50:11.393ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee file = illumos-utils/src/zone.rs:316
11042024-10-09T05:50:11.400ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 file = illumos-utils/src/zone.rs:354
11052024-10-09T05:50:11.409ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc file = illumos-utils/src/zone.rs:354
11062024-10-09T05:50:11.424ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc file = illumos-utils/src/zone.rs:354
11072024-10-09T05:50:11.434ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee file = illumos-utils/src/zone.rs:354
11082024-10-09T05:50:13.642ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40500
11092024-10-09T05:50:13.643ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 234 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40500 req_id = 30a7eacc-8e4d-4b3e-9274-3f702550b1c8 response_code = 200 uri = /baseboard
11102024-10-09T05:50:13.765ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_574723e3-b615-4bd5-bf94-f101903d419b/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='0e4ad732-091b-49ed-bb37-61eee121b051'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11112024-10-09T05:50:13.765ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
11122024-10-09T05:50:13.839ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_129ad649-4b7b-42ac-ac7c-60432cf5d7b0/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='1b0f7613-c24b-4775-869b-4ed4812dc4a9'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11132024-10-09T05:50:13.839ZINFOSledAgent (ServiceManager): Booting oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11142024-10-09T05:50:13.891ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_644e9307-d700-417a-856e-c71809b73b46: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f4492cbc-caa2-4fd8-bbf5-d062483d7508/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='2e0eb61e-45f6-4a68-9d32-2bdb419baf68'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11152024-10-09T05:50:13.891ZINFOSledAgent (ServiceManager): Booting oxz_crucible_644e9307-d700-417a-856e-c71809b73b46 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11162024-10-09T05:50:13.899ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11172024-10-09T05:50:13.899ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11182024-10-09T05:50:13.960ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f65f0473-59e4-4c49-bb65-a3d5f7f50633/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='e34d8030-eb0c-45c7-a2c7-28a622f9ee6e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11192024-10-09T05:50:13.960ZINFOSledAgent (ServiceManager): Booting oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11202024-10-09T05:50:14.045ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11212024-10-09T05:50:14.045ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11222024-10-09T05:50:14.104ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c4821672-b400-49f1-a3c1-f8485fe155ca/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='636c6747-1b2f-4eec-92a9-dda6385a3150'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11232024-10-09T05:50:14.105ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11242024-10-09T05:50:14.113ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_55ecf9f3-173f-4dac-bf98-2dda58cc9478/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='2930e4cc-9938-4105-a437-b20fea9ee2ce'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11252024-10-09T05:50:14.113ZINFOSledAgent (ServiceManager): Booting oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
11262024-10-09T05:50:14.254ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_fe683b07-b738-4a57-88f5-50899c5ceee2/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='0bcffbbd-a5de-47c9-8c11-655bbadb8550'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11272024-10-09T05:50:14.254ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11282024-10-09T05:50:14.270ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_8a5b3fe8-0ce2-44f3-ab61-dd61b9bf6628/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a9af60ae-6268-4d11-b8fb-65a92a066921'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11292024-10-09T05:50:14.271ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11302024-10-09T05:50:20.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 56.407038ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
11312024-10-09T05:50:20.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 25.22929ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11322024-10-09T05:50:20.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 34.687761ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11332024-10-09T05:50:20.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 71.438648ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11342024-10-09T05:50:20.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 46.378796ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
11352024-10-09T05:50:21.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 62.174857ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11362024-10-09T05:50:21.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 53.458228ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11372024-10-09T05:50:21.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 29.106893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11382024-10-09T05:50:21.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 93.326257ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
11392024-10-09T05:50:21.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 124.390173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11402024-10-09T05:50:21.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 129.911321ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11412024-10-09T05:50:21.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 108.103545ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11422024-10-09T05:50:21.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 131.542661ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
11432024-10-09T05:50:21.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 55.238337ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11442024-10-09T05:50:21.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 95.185445ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11452024-10-09T05:50:21.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 94.619683ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11462024-10-09T05:50:21.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 32.983783ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11472024-10-09T05:50:21.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 275.122769ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
11482024-10-09T05:50:21.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 278.475058ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11492024-10-09T05:50:21.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 115.781067ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11502024-10-09T05:50:21.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 202.633882ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11512024-10-09T05:50:21.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 283.067667ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11522024-10-09T05:50:21.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 251.696907ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11532024-10-09T05:50:21.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 103.021516ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
11542024-10-09T05:50:21.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 115.744347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11552024-10-09T05:50:21.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 260.346279ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11562024-10-09T05:50:21.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 253.695336ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11572024-10-09T05:50:21.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 66.647456ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11582024-10-09T05:50:21.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 589.12298ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
11592024-10-09T05:50:21.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 539.949657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11602024-10-09T05:50:21.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 79.054027ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11612024-10-09T05:50:21.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 550.433347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11622024-10-09T05:50:21.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 311.239726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
11632024-10-09T05:50:21.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 515.534872ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11642024-10-09T05:50:21.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 544.895499ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11652024-10-09T05:50:21.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 135.054162ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11662024-10-09T05:50:21.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 450.849105ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11672024-10-09T05:50:21.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 571.214668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11682024-10-09T05:50:21.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 275.079593ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11692024-10-09T05:50:21.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 269.37058ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11702024-10-09T05:50:21.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.198785138s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
11712024-10-09T05:50:21.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 625.206413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11722024-10-09T05:50:21.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 423.943304ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11732024-10-09T05:50:21.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 965.097657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11742024-10-09T05:50:21.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 876.949996ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
11752024-10-09T05:50:21.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 852.354823ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11762024-10-09T05:50:21.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 643.813043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11772024-10-09T05:50:21.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 823.827825ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11782024-10-09T05:50:22.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 715.228888ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11792024-10-09T05:50:22.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 437.498508ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11802024-10-09T05:50:22.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.2649246s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11812024-10-09T05:50:22.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.066453437s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11822024-10-09T05:50:22.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 647.164159ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11832024-10-09T05:50:22.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.170059531s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11842024-10-09T05:50:23.265ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 15.779893119s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
11852024-10-09T05:50:23.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.219009452s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
11862024-10-09T05:50:23.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.07085621s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11872024-10-09T05:50:23.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 514.427855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11882024-10-09T05:50:23.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.061782584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
11892024-10-09T05:50:23.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 908.159569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11902024-10-09T05:50:23.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 508.543886ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11912024-10-09T05:50:23.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.267722772s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
11922024-10-09T05:50:23.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.2056738s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
11932024-10-09T05:50:23.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 949.854332ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
11942024-10-09T05:50:23.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 946.941432ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
11952024-10-09T05:50:23.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.309648726s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
11962024-10-09T05:50:23.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.465942715s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
11972024-10-09T05:50:24.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 639.294423ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
11982024-10-09T05:50:24.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 740.911286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
11992024-10-09T05:50:24.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 627.935824ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12002024-10-09T05:50:24.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.300250214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
12012024-10-09T05:50:24.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 878.770707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
12022024-10-09T05:50:24.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 912.827735ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
12032024-10-09T05:50:24.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.047343694s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
12042024-10-09T05:50:24.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 746.272255ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
12052024-10-09T05:50:24.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.272602837s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
12062024-10-09T05:50:25.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 611.330817ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12072024-10-09T05:50:25.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.435888012s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
12082024-10-09T05:50:25.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.256409129s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
12092024-10-09T05:50:25.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.432384165s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
12102024-10-09T05:50:25.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.145012655s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
12112024-10-09T05:50:25.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 933.0891ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
12122024-10-09T05:50:25.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.24545718s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
12132024-10-09T05:50:25.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 880.973979ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12142024-10-09T05:50:25.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 584.529599ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
12152024-10-09T05:50:25.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 564.409393ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
12162024-10-09T05:50:26.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 554.09569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
12172024-10-09T05:50:26.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 701.865836ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
12182024-10-09T05:50:26.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.351820337s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
12192024-10-09T05:50:26.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 522.586078ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
12202024-10-09T05:50:26.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.420996785s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
12212024-10-09T05:50:26.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 811.468175ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12222024-10-09T05:50:26.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.435518106s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
12232024-10-09T05:50:26.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.042283086s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
12242024-10-09T05:50:26.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.144338681s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
12252024-10-09T05:50:26.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.022074932s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
12262024-10-09T05:50:26.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 771.090308ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
12272024-10-09T05:50:26.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.473644976s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
12282024-10-09T05:50:27.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 630.417775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
12292024-10-09T05:50:27.346ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12302024-10-09T05:50:27.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.192770112s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12312024-10-09T05:50:27.386ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12322024-10-09T05:50:27.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 702.532481ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
12332024-10-09T05:50:27.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 848.09847ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
12342024-10-09T05:50:27.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.202137054s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
12352024-10-09T05:50:27.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.107439661s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
12362024-10-09T05:50:27.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 62.557517ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12372024-10-09T05:50:27.874ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 851.227427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
12382024-10-09T05:50:27.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 57.851906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12392024-10-09T05:50:27.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 750.860236ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
12402024-10-09T05:50:28.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 562.529074ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
12412024-10-09T05:50:28.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 686.922917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
12422024-10-09T05:50:28.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 141.721996ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12432024-10-09T05:50:28.090ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='8b21fd27-df66-4b03-ac3d-86442491921a'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12442024-10-09T05:50:28.091ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12452024-10-09T05:50:28.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 209.521511ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12462024-10-09T05:50:28.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.313679672s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
12472024-10-09T05:50:28.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 58.3655ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12482024-10-09T05:50:28.443ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_6d483f9a-1428-49ad-be75-fa82682b3fb8/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='c4ce77de-6139-4c5d-b547-eead234fc056'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12492024-10-09T05:50:28.444ZINFOSledAgent (ServiceManager): Booting oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
12502024-10-09T05:50:28.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.394740131s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
12512024-10-09T05:50:28.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 65.53181ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12522024-10-09T05:50:28.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 464.462361ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12532024-10-09T05:50:28.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 248.836865ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12542024-10-09T05:50:28.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 997.847964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
12552024-10-09T05:50:28.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.134969377s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12562024-10-09T05:50:28.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 866.237874ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
12572024-10-09T05:50:28.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.465268749s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
12582024-10-09T05:50:28.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 953.261323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
12592024-10-09T05:50:28.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 51.365477ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
12602024-10-09T05:50:28.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 662.823713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
12612024-10-09T05:50:28.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 98.774168ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
12622024-10-09T05:50:28.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 206.779302ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12632024-10-09T05:50:28.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 925.249196ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
12642024-10-09T05:50:28.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 191.690912ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
12652024-10-09T05:50:28.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.001869082s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12662024-10-09T05:50:29.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 554.13203ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
12672024-10-09T05:50:29.078ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 539.612561ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12682024-10-09T05:50:29.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 455.316858ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
12692024-10-09T05:50:29.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.417808225s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
12702024-10-09T05:50:29.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 742.482508ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
12712024-10-09T05:50:29.532ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_94415647-751f-4321-816f-7e4752b52496: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12722024-10-09T05:50:29.533ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_94415647-751f-4321-816f-7e4752b52496 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
12732024-10-09T05:50:29.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.471197748s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
12742024-10-09T05:50:29.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.147989248s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
12752024-10-09T05:50:29.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.29436949s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12762024-10-09T05:50:29.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.185305703s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
12772024-10-09T05:50:29.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.382927875s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
12782024-10-09T05:50:29.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 966.408065ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
12792024-10-09T05:50:29.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 620.634779ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12802024-10-09T05:50:29.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 64.235786ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
12812024-10-09T05:50:29.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 74.462119ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
12822024-10-09T05:50:29.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 620.829342ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
12832024-10-09T05:50:29.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.078575133s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
12842024-10-09T05:50:29.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 279.564807ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
12852024-10-09T05:50:30.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.238499416s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
12862024-10-09T05:50:30.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 731.572293ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
12872024-10-09T05:50:30.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 514.591334ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
12882024-10-09T05:50:30.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 714.816419ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
12892024-10-09T05:50:30.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 971.296548ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
12902024-10-09T05:50:30.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.354014179s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
12912024-10-09T05:50:30.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.050017898s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
12922024-10-09T05:50:30.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 864.342449ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
12932024-10-09T05:50:30.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.070444925s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
12942024-10-09T05:50:30.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 902.676094ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
12952024-10-09T05:50:30.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.08447937s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
12962024-10-09T05:50:30.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 586.714314ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
12972024-10-09T05:50:30.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 639.605208ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
12982024-10-09T05:50:30.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.464455394s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
12992024-10-09T05:50:31.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.235068831s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
13002024-10-09T05:50:31.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 526.975537ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13012024-10-09T05:50:31.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 616.868295ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
13022024-10-09T05:50:31.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.318600001s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
13032024-10-09T05:50:31.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.245149524s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
13042024-10-09T05:50:31.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.276346066s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
13052024-10-09T05:50:31.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 601.044153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13062024-10-09T05:50:31.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.278439508s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
13072024-10-09T05:50:31.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 645.733418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
13082024-10-09T05:50:31.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.124933722s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
13092024-10-09T05:50:31.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.269234518s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
13102024-10-09T05:50:31.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 730.664543ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
13112024-10-09T05:50:31.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 884.124693ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
13122024-10-09T05:50:31.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 886.986975ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
13132024-10-09T05:50:32.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 877.81694ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
13142024-10-09T05:50:32.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 858.169085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13152024-10-09T05:50:32.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 839.005118ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
13162024-10-09T05:50:32.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.354518804s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
13172024-10-09T05:50:32.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.199579496s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
13182024-10-09T05:50:32.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 531.542068ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
13192024-10-09T05:50:32.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 566.82764ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
13202024-10-09T05:50:32.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.267329073s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
13212024-10-09T05:50:32.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 704.034078ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
13222024-10-09T05:50:32.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.162586457s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
13232024-10-09T05:50:32.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 762.356284ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
13242024-10-09T05:50:32.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.317669289s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
13252024-10-09T05:50:32.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 951.740063ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
13262024-10-09T05:50:32.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 618.884527ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
13272024-10-09T05:50:33.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.19895971s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
13282024-10-09T05:50:33.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 856.26235ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
13292024-10-09T05:50:33.077ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 524.115989ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13302024-10-09T05:50:33.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.05708948s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
13312024-10-09T05:50:33.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 830.366287ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
13322024-10-09T05:50:33.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.100952849s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
13332024-10-09T05:50:33.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.128996352s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
13342024-10-09T05:50:33.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 812.006222ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
13352024-10-09T05:50:33.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 964.161026ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13362024-10-09T05:50:33.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 528.762426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
13372024-10-09T05:50:33.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.180047127s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
13382024-10-09T05:50:33.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 980.484742ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
13392024-10-09T05:50:33.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 508.496046ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
13402024-10-09T05:50:33.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.071426321s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
13412024-10-09T05:50:34.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 559.037542ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
13422024-10-09T05:50:34.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 811.784775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
13432024-10-09T05:50:34.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 979.601237ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
13442024-10-09T05:50:34.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 550.118322ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
13452024-10-09T05:50:34.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.066199987s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
13462024-10-09T05:50:34.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 993.312077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
13472024-10-09T05:50:34.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.270641566s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
13482024-10-09T05:50:34.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.405376754s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
13492024-10-09T05:50:34.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.234879144s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
13502024-10-09T05:50:34.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.055903009s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13512024-10-09T05:50:34.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.386119849s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
13522024-10-09T05:50:34.609ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13532024-10-09T05:50:34.609ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
13542024-10-09T05:50:34.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 611.725102ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
13552024-10-09T05:50:34.729ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13562024-10-09T05:50:34.730ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13572024-10-09T05:50:34.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.124537009s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
13582024-10-09T05:50:34.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 41.035652ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
13592024-10-09T05:50:34.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 68.659222ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
13602024-10-09T05:50:34.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.471682668s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
13612024-10-09T05:50:34.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 831.15898ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
13622024-10-09T05:50:34.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 138.49807ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
13632024-10-09T05:50:34.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 669.632552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
13642024-10-09T05:50:34.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 28.954953ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13652024-10-09T05:50:34.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 75.256045ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13662024-10-09T05:50:35.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 768.579246ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
13672024-10-09T05:50:35.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 340.54061ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
13682024-10-09T05:50:35.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 162.745099ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13692024-10-09T05:50:35.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 600.367351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
13702024-10-09T05:50:35.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 392.55375ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13712024-10-09T05:50:35.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.363300908s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
13722024-10-09T05:50:35.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 792.281995ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
13732024-10-09T05:50:35.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.461259362s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
13742024-10-09T05:50:35.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.115704089s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
13752024-10-09T05:50:35.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 671.141205ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
13762024-10-09T05:50:35.653ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 657.663693ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13772024-10-09T05:50:35.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 792.879871ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13782024-10-09T05:50:35.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.333388911s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
13792024-10-09T05:50:35.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.143865161s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
13802024-10-09T05:50:35.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.009103189s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
13812024-10-09T05:50:35.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 693.303521ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
13822024-10-09T05:50:35.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 656.362474ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
13832024-10-09T05:50:35.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.475450878s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
13842024-10-09T05:50:35.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.483740937s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
13852024-10-09T05:50:35.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 984.484464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
13862024-10-09T05:50:36.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.414489155s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
13872024-10-09T05:50:36.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 758.865977ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
13882024-10-09T05:50:36.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 628.924578ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13892024-10-09T05:50:36.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 892.945013ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
13902024-10-09T05:50:36.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.202966483s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
13912024-10-09T05:50:36.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.108270695s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
13922024-10-09T05:50:36.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 685.381148ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
13932024-10-09T05:50:36.541ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.437484119s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
13942024-10-09T05:50:36.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 735.834967ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
13952024-10-09T05:50:36.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.111387124s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
13962024-10-09T05:50:36.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 989.946575ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
13972024-10-09T05:50:36.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.062752976s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
13982024-10-09T05:50:36.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 645.323296ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
13992024-10-09T05:50:36.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 624.678497ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
14002024-10-09T05:50:37.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.340710028s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
14012024-10-09T05:50:37.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 895.853693ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
14022024-10-09T05:50:37.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 953.664676ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
14032024-10-09T05:50:37.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.2026842s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
14042024-10-09T05:50:37.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.008306564s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
14052024-10-09T05:50:37.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.337572387s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
14062024-10-09T05:50:37.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.124937349s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
14072024-10-09T05:50:37.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.404599152s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
14082024-10-09T05:50:37.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 825.266139ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
14092024-10-09T05:50:37.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 942.210481ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
14102024-10-09T05:50:37.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 796.191977ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
14112024-10-09T05:50:37.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.071223422s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
14122024-10-09T05:50:37.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.072161654s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
14132024-10-09T05:50:37.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.441238284s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
14142024-10-09T05:50:37.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.142500114s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
14152024-10-09T05:50:37.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 707.45545ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
14162024-10-09T05:50:37.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.37644772s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
14172024-10-09T05:50:38.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 802.848518ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
14182024-10-09T05:50:38.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.263497677s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
14192024-10-09T05:50:38.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 842.818498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
14202024-10-09T05:50:38.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.1859427s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
14212024-10-09T05:50:38.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.146802949s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
14222024-10-09T05:50:38.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.309828425s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
14232024-10-09T05:50:38.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 692.398534ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
14242024-10-09T05:50:38.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 968.546648ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
14252024-10-09T05:50:38.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 943.962426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
14262024-10-09T05:50:38.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 697.51824ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
14272024-10-09T05:50:38.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 876.767871ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
14282024-10-09T05:50:38.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.080658444s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
14292024-10-09T05:50:38.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.440378386s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
14302024-10-09T05:50:39.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 554.894921ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
14312024-10-09T05:50:39.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.450296172s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
14322024-10-09T05:50:39.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 724.896684ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
14332024-10-09T05:50:39.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.087279465s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
14342024-10-09T05:50:39.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.309670367s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
14352024-10-09T05:50:39.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 876.778974ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
14362024-10-09T05:50:39.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.398271343s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
14372024-10-09T05:50:39.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 539.066998ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
14382024-10-09T05:50:39.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 655.366564ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
14392024-10-09T05:50:39.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 733.482983ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
14402024-10-09T05:50:39.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 851.239445ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
14412024-10-09T05:50:39.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.348241334s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
14422024-10-09T05:50:39.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.15406308s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
14432024-10-09T05:50:39.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.432893157s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
14442024-10-09T05:50:39.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.00963529s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
14452024-10-09T05:50:39.843ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14462024-10-09T05:50:39.843ZINFOSledAgent (ServiceManager): Booting oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
14472024-10-09T05:50:39.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 550.205542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
14482024-10-09T05:50:39.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.423753889s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
14492024-10-09T05:50:39.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 52.608305ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
14502024-10-09T05:50:40.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 106.382026ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
14512024-10-09T05:50:40.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 1.054495233s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
14522024-10-09T05:50:40.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 215.842229ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
14532024-10-09T05:50:40.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 616.198353ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
14542024-10-09T05:50:40.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 959.208076ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
14552024-10-09T05:50:40.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 586.65137ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
14562024-10-09T05:50:40.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 716.724138ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
14572024-10-09T05:50:40.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.054368337s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
14582024-10-09T05:50:40.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 316.860032ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
14592024-10-09T05:50:40.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 570.294182ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
14602024-10-09T05:50:40.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.113136558s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
14612024-10-09T05:50:40.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.13581434s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
14622024-10-09T05:50:40.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 705.805418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
14632024-10-09T05:50:40.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 1.085807335s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
14642024-10-09T05:50:40.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.219787772s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
14652024-10-09T05:50:40.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.317157055s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
14662024-10-09T05:50:40.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 669.100286ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
14672024-10-09T05:50:40.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.036183078s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
14682024-10-09T05:50:40.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 817.303269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
14692024-10-09T05:50:41.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 666.605603ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
14702024-10-09T05:50:41.012ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14712024-10-09T05:50:41.012ZINFOSledAgent (ServiceManager): Booting oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
14722024-10-09T05:50:41.018ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14732024-10-09T05:50:41.018ZINFOSledAgent (ServiceManager): Booting oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
14742024-10-09T05:50:41.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 747.257147ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
14752024-10-09T05:50:41.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 970.225664ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
14762024-10-09T05:50:41.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 965.182957ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
14772024-10-09T05:50:41.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 1.473807063s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
14782024-10-09T05:50:41.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 53.978689ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
14792024-10-09T05:50:41.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 57.415437ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
14802024-10-09T05:50:41.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 657.989227ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
14812024-10-09T05:50:41.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 56.463337ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
14822024-10-09T05:50:41.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 144.174345ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
14832024-10-09T05:50:41.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 214.717055ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
14842024-10-09T05:50:41.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 848.966669ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
14852024-10-09T05:50:41.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.207315594s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
14862024-10-09T05:50:41.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 145.927488ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
14872024-10-09T05:50:41.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.490162911s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
14882024-10-09T05:50:41.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.299314829s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
14892024-10-09T05:50:41.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 310.774217ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
14902024-10-09T05:50:41.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 373.081595ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
14912024-10-09T05:50:41.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.029285671s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
14922024-10-09T05:50:41.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 685.149595ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
14932024-10-09T05:50:41.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.479145021s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
14942024-10-09T05:50:41.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 932.805561ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
14952024-10-09T05:50:41.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 1.312015445s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
14962024-10-09T05:50:41.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.243430758s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
14972024-10-09T05:50:41.909ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.060607993s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
14982024-10-09T05:50:41.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 937.105706ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
14992024-10-09T05:50:41.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 661.495727ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
15002024-10-09T05:50:41.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 647.987807ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
15012024-10-09T05:50:42.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.304743112s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
15022024-10-09T05:50:42.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.065701024s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
15032024-10-09T05:50:42.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 788.49913ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
15042024-10-09T05:50:42.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.3933877s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
15052024-10-09T05:50:42.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.380468795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
15062024-10-09T05:50:42.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 890.286779ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
15072024-10-09T05:50:42.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 604.503738ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
15082024-10-09T05:50:42.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 592.440974ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
15092024-10-09T05:50:42.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.498812818s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
15102024-10-09T05:50:42.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 1.429099357s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
15112024-10-09T05:50:42.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.402992614s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
15122024-10-09T05:50:42.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 686.238155ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
15132024-10-09T05:50:42.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 673.899063ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
15142024-10-09T05:50:42.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 556.672417ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
15152024-10-09T05:50:42.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 1.47299339s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
15162024-10-09T05:50:42.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 841.044196ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
15172024-10-09T05:50:42.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.338594133s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
15182024-10-09T05:50:43.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 674.765746ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
15192024-10-09T05:50:43.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 722.538609ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
15202024-10-09T05:50:43.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.443465811s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
15212024-10-09T05:50:43.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.147791133s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
15222024-10-09T05:50:43.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.459125277s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
15232024-10-09T05:50:43.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 1.326475327s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
15242024-10-09T05:50:43.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.49712679s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
15252024-10-09T05:50:43.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 797.643855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
15262024-10-09T05:50:43.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.350360485s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
15272024-10-09T05:50:43.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 829.490524ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
15282024-10-09T05:50:43.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 505.925542ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
15292024-10-09T05:50:43.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 615.351493ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
15302024-10-09T05:50:43.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 805.218773ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
15312024-10-09T05:50:43.719ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59354
15322024-10-09T05:50:43.725ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3541 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59354 req_id = 70ce9690-d9e6-49b1-8834-fc335561f400 response_code = 200 uri = /baseboard
15332024-10-09T05:50:43.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.143721564s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
15342024-10-09T05:50:43.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 1.429632036s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
15352024-10-09T05:50:43.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 1.039345562s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
15362024-10-09T05:50:44.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.038481867s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
15372024-10-09T05:50:44.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 580.797738ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
15382024-10-09T05:50:44.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 893.677851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
15392024-10-09T05:50:44.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.435150221s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
15402024-10-09T05:50:44.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.029649032s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
15412024-10-09T05:50:44.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.367681553s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
15422024-10-09T05:50:44.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 1.161411954s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
15432024-10-09T05:50:44.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 874.209262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
15442024-10-09T05:50:44.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 730.847827ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
15452024-10-09T05:50:44.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 901.870912ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
15462024-10-09T05:50:44.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.175958268s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
15472024-10-09T05:50:44.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 707.215844ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
15482024-10-09T05:50:44.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 800.902502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
15492024-10-09T05:50:44.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 609.592227ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
15502024-10-09T05:50:44.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.107568353s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
15512024-10-09T05:50:44.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.11818089s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
15522024-10-09T05:50:44.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.324086613s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
15532024-10-09T05:50:44.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 1.074062502s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
15542024-10-09T05:50:44.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.107975342s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
15552024-10-09T05:50:45.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.119166553s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
15562024-10-09T05:50:45.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 831.299179ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
15572024-10-09T05:50:45.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 842.356013ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
15582024-10-09T05:50:45.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 617.393271ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
15592024-10-09T05:50:45.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.089461666s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
15602024-10-09T05:50:45.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 983.78182ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
15612024-10-09T05:50:45.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 620.654601ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
15622024-10-09T05:50:45.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 874.495261ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
15632024-10-09T05:50:45.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 1.439403446s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
15642024-10-09T05:50:45.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.374825907s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
15652024-10-09T05:50:45.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 578.948406ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
15662024-10-09T05:50:45.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 896.767525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
15672024-10-09T05:50:45.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416") failed: Property not found. retry in 1.091223337s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
15682024-10-09T05:50:45.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d") failed: Property not found. retry in 1.069030348s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
15692024-10-09T05:50:45.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 671.411699ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
15702024-10-09T05:50:45.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235") failed: Property not found. retry in 1.19532655s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
15712024-10-09T05:50:45.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2") failed: Property not found. retry in 1.234022241s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
15722024-10-09T05:50:45.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 923.534459ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
15732024-10-09T05:50:45.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 653.052593ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
15742024-10-09T05:50:45.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554") failed: Property not found. retry in 1.474457382s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
15752024-10-09T05:50:46.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 919.699483ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
15762024-10-09T05:50:46.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 1.375017212s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
15772024-10-09T05:50:46.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 904.532422ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
15782024-10-09T05:50:46.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.43550124s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
15792024-10-09T05:50:46.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 632.663106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
15802024-10-09T05:50:46.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 776.604518ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
15812024-10-09T05:50:46.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a") failed: Property not found. retry in 975.313959ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
15822024-10-09T05:50:46.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_644e9307-d700-417a-856e-c71809b73b46") failed: Property not found. retry in 1.345449994s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
15832024-10-09T05:50:46.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.458976963s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
15842024-10-09T05:50:46.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.266120831s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
15852024-10-09T05:50:46.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 941.453291ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
15862024-10-09T05:50:46.770ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 1.044323639s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
15872024-10-09T05:50:46.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 674.148813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
15882024-10-09T05:50:46.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.464166669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
15892024-10-09T05:50:46.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de") failed: Property not found. retry in 505.77915ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
15902024-10-09T05:50:46.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 780.891368ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
15912024-10-09T05:50:46.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.098820961s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
15922024-10-09T05:50:46.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 1.340444435s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
15932024-10-09T05:50:47.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 1.129481169s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
15942024-10-09T05:50:47.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.40195323s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
15952024-10-09T05:50:47.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 1.206125915s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
15962024-10-09T05:50:47.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 909.992896ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
15972024-10-09T05:50:47.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 1.040297512s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
15982024-10-09T05:50:47.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 642.768739ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
15992024-10-09T05:50:47.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 808.744138ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
16002024-10-09T05:50:47.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 703.285642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
16012024-10-09T05:50:48.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 657.430895ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
16022024-10-09T05:50:48.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 911.55402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
16032024-10-09T05:50:48.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 757.86801ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
16042024-10-09T05:50:48.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.093733076s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
16052024-10-09T05:50:48.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 651.208546ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
16062024-10-09T05:50:48.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 1.261920863s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
16072024-10-09T05:50:48.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 866.226584ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
16082024-10-09T05:50:48.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 755.844372ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
16092024-10-09T05:50:48.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 535.107547ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
16102024-10-09T05:50:48.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a") failed: Property not found. retry in 844.460062ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
16112024-10-09T05:50:48.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 843.086416ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
16122024-10-09T05:50:48.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 869.185512ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
16132024-10-09T05:50:49.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 1.160455782s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
16142024-10-09T05:50:49.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797") failed: Property not found. retry in 1.347948488s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
16152024-10-09T05:50:49.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 795.697701ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
16162024-10-09T05:50:49.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_94415647-751f-4321-816f-7e4752b52496") failed: Property not found. retry in 1.281096609s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
16172024-10-09T05:50:49.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664") failed: Property not found. retry in 1.0130483s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
16182024-10-09T05:50:49.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 693.093849ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
16192024-10-09T05:50:49.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 667.51005ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
16202024-10-09T05:50:49.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 641.666244ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
16212024-10-09T05:50:49.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8") failed: Property not found. retry in 1.387786582s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
16222024-10-09T05:50:49.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee") failed: Property not found. retry in 1.394086216s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
16232024-10-09T05:50:50.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d") failed: Property not found. retry in 1.276111976s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
16242024-10-09T05:50:50.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 670.434857ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
16252024-10-09T05:50:50.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc") failed: Property not found. retry in 981.720143ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
16262024-10-09T05:50:50.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1") failed: Property not found. retry in 1.421517703s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
16272024-10-09T05:50:50.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc") failed: Property not found. retry in 1.05167341s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
16282024-10-09T05:50:50.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59") failed: Property not found. retry in 1.053234774s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
16292024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_f6002c58-4bdb-4ab2-a759-d7920f1c1bc2
16302024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_dcb46d89-108c-4535-bdf4-4263a2f94554
16312024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_oximeter_8b21fd27-df66-4b03-ac3d-86442491921a
16322024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_644e9307-d700-417a-856e-c71809b73b46
16332024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_external_dns_94415647-751f-4321-816f-7e4752b52496
16342024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_nexus_9a7d69b2-b5cb-4929-9d3e-5c8730f41abc
16352024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_nexus_a9346314-f5a8-4da4-8c60-48c5ab6c3dee
16362024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_clickhouse_9591b96e-20a2-403d-a1c3-ac848f6a8ae1
16372024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_pantry_74b33d8f-4ccf-4b0a-a237-609d67e5f664
16382024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_pantry_44f1da2d-5239-4bd4-9324-c1144628f93d
16392024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_bea743e6-cd46-4ddc-84f1-3d8e99a5995a
16402024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_4db41e00-66ff-487e-8c30-c084b43ec6de
16412024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_f4690efb-90e0-497f-9635-de08dea99416
16422024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_external_dns_d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59
16432024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_a0bf7df0-ebbb-414f-a736-03b7e818e235
16442024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_f7b6d332-9837-40d3-96a7-e562f6f28797
16452024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_pantry_f85f5cb6-5308-4cab-9f33-fd8db8cdc51d
16462024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_nexus_bcd392d4-4bfb-4d15-99aa-1265e84bc8dc
16472024-10-09T05:50:52.006ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3308 zone = oxz_crucible_11ca442c-f019-48a8-a842-a0f49e30b7a8
16482024-10-09T05:50:52.007ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16492024-10-09T05:50:52.024ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16502024-10-09T05:50:52.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 44291343 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37220 req_id = 886db727-9a34-4e23-a61d-85cc8949b28e response_code = 204 uri = /omicron-zones
16512024-10-09T05:50:52.048ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1322
16522024-10-09T05:50:52.048ZINFOSledAgent (RSS): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
16532024-10-09T05:50:52.049ZINFOSledAgent (RSS): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
16542024-10-09T05:50:52.058ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
16552024-10-09T05:50:52.059ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:720
16562024-10-09T05:50:52.094ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, rib_priority: None, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:887
16572024-10-09T05:50:52.096ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16582024-10-09T05:50:52.217ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16592024-10-09T05:50:52.286ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16602024-10-09T05:50:52.434ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16612024-10-09T05:50:52.591ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16622024-10-09T05:50:52.768ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16632024-10-09T05:50:52.951ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16642024-10-09T05:50:53.110ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16652024-10-09T05:50:53.478ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16662024-10-09T05:50:53.791ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16672024-10-09T05:50:54.163ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 40.940686007s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
16682024-10-09T05:50:54.459ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16692024-10-09T05:50:55.379ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16702024-10-09T05:50:56.313ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16712024-10-09T05:50:57.621ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16722024-10-09T05:50:58.285ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16732024-10-09T05:50:59.889ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16742024-10-09T05:51:01.698ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/cde79127-ad1a-4dc8-8bde-c617fda5df2f/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
16752024-10-09T05:51:03.624ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "bd5a2651-5571-4f03-beb8-eedadd651347", "content-length": "173", "date": "Wed, 09 Oct 2024 05:51:03 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"cde79127-ad1a-4dc8-8bde-c617fda5df2f\"", request_id: "bd5a2651-5571-4f03-beb8-eedadd651347" } file = sled-agent/src/rack_setup/service.rs:949
16762024-10-09T05:51:05.499ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "936fb9a3-ad4d-42d7-bcdd-16ab8dc89d27", "content-length": "173", "date": "Wed, 09 Oct 2024 05:51:05 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"cde79127-ad1a-4dc8-8bde-c617fda5df2f\"", request_id: "936fb9a3-ad4d-42d7-bcdd-16ab8dc89d27" } file = sled-agent/src/rack_setup/service.rs:949
16772024-10-09T05:51:07.997ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "6b16aaad-cf0f-4122-97d8-d63b0e5eae33", "content-length": "173", "date": "Wed, 09 Oct 2024 05:51:07 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"8fa2b36a-fea8-4835-8541-6d298f48930d\"", request_id: "6b16aaad-cf0f-4122-97d8-d63b0e5eae33" } file = sled-agent/src/rack_setup/service.rs:949
16782024-10-09T05:51:12.136ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "42acf753-f448-4945-8875-284f56fc75d9", "content-length": "173", "date": "Wed, 09 Oct 2024 05:51:11 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"8fa2b36a-fea8-4835-8541-6d298f48930d\"", request_id: "42acf753-f448-4945-8875-284f56fc75d9" } file = sled-agent/src/rack_setup/service.rs:949
16792024-10-09T05:51:13.773ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53962
16802024-10-09T05:51:13.773ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53962 req_id = 0022f8f8-cda1-4fad-9151-0fd0a82c6d35 response_code = 200 uri = /baseboard
16812024-10-09T05:51:17.914ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "6249b3f2-84ac-491e-82f6-5369c7c9e573", "content-length": "173", "date": "Wed, 09 Oct 2024 05:51:17 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"8fa2b36a-fea8-4835-8541-6d298f48930d\"", request_id: "6249b3f2-84ac-491e-82f6-5369c7c9e573" } file = sled-agent/src/rack_setup/service.rs:949
16822024-10-09T05:51:23.055ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "f19fa00d-e452-4583-b76c-455f617d7012", "content-length": "173", "date": "Wed, 09 Oct 2024 05:51:23 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"8fa2b36a-fea8-4835-8541-6d298f48930d\"", request_id: "f19fa00d-e452-4583-b76c-455f617d7012" } file = sled-agent/src/rack_setup/service.rs:949
16832024-10-09T05:51:29.151ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "3d8a99c0-5916-48e2-a4aa-204bf5f2731f", "content-length": "173", "date": "Wed, 09 Oct 2024 05:51:28 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"8fa2b36a-fea8-4835-8541-6d298f48930d\"", request_id: "3d8a99c0-5916-48e2-a4aa-204bf5f2731f" } file = sled-agent/src/rack_setup/service.rs:949
16842024-10-09T05:51:33.295ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
16852024-10-09T05:51:33.392ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61534
16862024-10-09T05:51:33.394ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16872024-10-09T05:51:33.592ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 199628 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61534 req_id = 58efac11-a4d4-4bd7-9160-9f631dc34418 response_code = 200 uri = /inventory
16882024-10-09T05:51:33.594ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16892024-10-09T05:51:33.594ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-zones.json file = common/src/ledger.rs:177
16902024-10-09T05:51:33.595ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-zones.json file = common/src/ledger.rs:177
16912024-10-09T05:51:33.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2377 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61534 req_id = 797dfdc3-1735-4c2b-829f-6b9a4a3a73ed response_code = 200 uri = /omicron-zones
16922024-10-09T05:51:34.049ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47464
16932024-10-09T05:51:34.052ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 653f6089-0158-484c-84b9-bfd9e4d86b4f, kind: Service { id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 }, name: Name("external-dns-d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 211, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cad5ed0d-3ea9-489d-93d4-7e3565b87eb6, kind: Service { id: 94415647-751f-4321-816f-7e4752b52496 }, name: Name("external-dns-94415647-751f-4321-816f-7e4752b52496"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 7])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 08db3328-de2f-4bc0-aa29-88948e0b323e, kind: Service { id: a9346314-f5a8-4da4-8c60-48c5ab6c3dee }, name: Name("nexus-a9346314-f5a8-4da4-8c60-48c5ab6c3dee"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6296d238-bc00-44df-a0e2-f8ef31c024da, kind: Service { id: bcd392d4-4bfb-4d15-99aa-1265e84bc8dc }, name: Name("nexus-bcd392d4-4bfb-4d15-99aa-1265e84bc8dc"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 191, 204])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f2385eb-ba1c-4d5f-9e21-61e91356244e, kind: Service { id: 9a7d69b2-b5cb-4929-9d3e-5c8730f41abc }, name: Name("nexus-9a7d69b2-b5cb-4929-9d3e-5c8730f41abc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 96])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16942024-10-09T05:51:34.052ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16952024-10-09T05:51:34.053ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16962024-10-09T05:51:34.053ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16972024-10-09T05:51:34.053ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
16982024-10-09T05:51:34.053ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16992024-10-09T05:51:34.053ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17002024-10-09T05:51:34.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3239 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47464 req_id = 15f9285d-1e2f-4b94-b672-814d92ba30d5 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17012024-10-09T05:51:34.053ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:959
17022024-10-09T05:51:34.280ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38715
17032024-10-09T05:51:34.280ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17042024-10-09T05:51:34.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 185867 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38715 req_id = 91e9351b-b999-4316-acdc-cd5b102ea855 response_code = 200 uri = /inventory
17052024-10-09T05:51:34.469ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17062024-10-09T05:51:34.469ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/config/omicron-zones.json file = common/src/ledger.rs:177
17072024-10-09T05:51:34.469ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/config/omicron-zones.json file = common/src/ledger.rs:177
17082024-10-09T05:51:34.470ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1542 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38715 req_id = 3b41c899-3071-4e01-8e2c-f39d5b1a2d7a response_code = 200 uri = /omicron-zones
17092024-10-09T05:51:34.958ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49008
17102024-10-09T05:51:34.958ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 653f6089-0158-484c-84b9-bfd9e4d86b4f, kind: Service { id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 }, name: Name("external-dns-d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 211, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cad5ed0d-3ea9-489d-93d4-7e3565b87eb6, kind: Service { id: 94415647-751f-4321-816f-7e4752b52496 }, name: Name("external-dns-94415647-751f-4321-816f-7e4752b52496"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 7])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 08db3328-de2f-4bc0-aa29-88948e0b323e, kind: Service { id: a9346314-f5a8-4da4-8c60-48c5ab6c3dee }, name: Name("nexus-a9346314-f5a8-4da4-8c60-48c5ab6c3dee"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6296d238-bc00-44df-a0e2-f8ef31c024da, kind: Service { id: bcd392d4-4bfb-4d15-99aa-1265e84bc8dc }, name: Name("nexus-bcd392d4-4bfb-4d15-99aa-1265e84bc8dc"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 191, 204])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f2385eb-ba1c-4d5f-9e21-61e91356244e, kind: Service { id: 9a7d69b2-b5cb-4929-9d3e-5c8730f41abc }, name: Name("nexus-9a7d69b2-b5cb-4929-9d3e-5c8730f41abc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 96])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17112024-10-09T05:51:34.958ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17122024-10-09T05:51:34.958ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17132024-10-09T05:51:34.958ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17142024-10-09T05:51:34.958ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
17152024-10-09T05:51:34.958ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17162024-10-09T05:51:34.958ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17172024-10-09T05:51:34.958ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 307 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49008 req_id = 2ea70a59-51fc-4ecc-a026-17341bafb4cf response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17182024-10-09T05:51:35.044ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63068
17192024-10-09T05:51:35.044ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 653f6089-0158-484c-84b9-bfd9e4d86b4f, kind: Service { id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 }, name: Name("external-dns-d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 211, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cad5ed0d-3ea9-489d-93d4-7e3565b87eb6, kind: Service { id: 94415647-751f-4321-816f-7e4752b52496 }, name: Name("external-dns-94415647-751f-4321-816f-7e4752b52496"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 7])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 08db3328-de2f-4bc0-aa29-88948e0b323e, kind: Service { id: a9346314-f5a8-4da4-8c60-48c5ab6c3dee }, name: Name("nexus-a9346314-f5a8-4da4-8c60-48c5ab6c3dee"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6296d238-bc00-44df-a0e2-f8ef31c024da, kind: Service { id: bcd392d4-4bfb-4d15-99aa-1265e84bc8dc }, name: Name("nexus-bcd392d4-4bfb-4d15-99aa-1265e84bc8dc"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 191, 204])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f2385eb-ba1c-4d5f-9e21-61e91356244e, kind: Service { id: 9a7d69b2-b5cb-4929-9d3e-5c8730f41abc }, name: Name("nexus-9a7d69b2-b5cb-4929-9d3e-5c8730f41abc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 96])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17202024-10-09T05:51:35.044ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17212024-10-09T05:51:35.044ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17222024-10-09T05:51:35.044ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17232024-10-09T05:51:35.044ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
17242024-10-09T05:51:35.044ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17252024-10-09T05:51:35.044ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17262024-10-09T05:51:35.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 246 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63068 req_id = 65deb2cd-f6c4-47bd-a1f9-5346f4b0607a response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17272024-10-09T05:51:35.236ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38107
17282024-10-09T05:51:35.236ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 271 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38107 req_id = a8eaff0b-eca2-4f70-987c-ac1cfc680690 response_code = 200 uri = /vpc-routes
17292024-10-09T05:51:35.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34685
17302024-10-09T05:51:35.262ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34685 req_id = c61e6cae-4b77-4f2d-9566-79bba8ed3597 response_code = 200 uri = /vpc-routes
17312024-10-09T05:51:35.270ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
17322024-10-09T05:51:35.270ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 882 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38107 req_id = d4dea14a-6dca-4a81-a632-048620efce20 response_code = 204 uri = /eip-gateways
17332024-10-09T05:51:35.388ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
17342024-10-09T05:51:35.388ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34685 req_id = 3f9f023c-9c38-4742-a167-a647f5d3a25e response_code = 204 uri = /eip-gateways
17352024-10-09T05:51:35.512ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44465
17362024-10-09T05:51:35.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44465 req_id = 24c7c311-35eb-4652-be54-c9c2ae27c2cb response_code = 200 uri = /v2p
17372024-10-09T05:51:35.516ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34992
17382024-10-09T05:51:35.517ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
17392024-10-09T05:51:35.612ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
17402024-10-09T05:51:35.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 175523 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34992 req_id = ef81fa41-5fea-46ed-bfb4-124b5fed4893 response_code = 204 uri = /switch-ports
17412024-10-09T05:51:35.695ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3086 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34992 req_id = 31c62566-a5bd-442c-ae7b-74ffc311bb7e response_code = 200 uri = /network-bootstore-config
17422024-10-09T05:51:35.695ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59323
17432024-10-09T05:51:35.696ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
17442024-10-09T05:51:35.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8986 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34685 req_id = e9e96926-e8e0-4dac-90bc-04e73ae051bf response_code = 204 uri = /vpc-routes
17452024-10-09T05:51:35.719ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
17462024-10-09T05:51:35.719ZINFOSledAgent (bootstore): Writing ledger to /pool/int/d6cde337-7634-4a9e-a6ab-bc764b121353/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
17472024-10-09T05:51:35.721ZINFOSledAgent (bootstore): Writing ledger to /pool/int/07b7cea6-020b-4d1f-aaaf-8237f5cf7a8c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
17482024-10-09T05:51:35.722ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
17492024-10-09T05:51:35.722ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4569 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34992 req_id = dc857070-6fd9-4839-a9b1-713777bceffb response_code = 204 uri = /network-bootstore-config
17502024-10-09T05:51:35.745ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
17512024-10-09T05:51:35.811ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45355
17522024-10-09T05:51:35.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 115573 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59323 req_id = 795969c3-fb23-422f-8450-87bd51110ab0 response_code = 204 uri = /switch-ports
17532024-10-09T05:51:35.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 215 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45355 req_id = abcbc7ac-68cf-4d35-88f4-8fabce1ace0c response_code = 200 uri = /v2p
17542024-10-09T05:51:35.812ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59323 req_id = a6b58b4c-14e9-40ae-a610-71c6fc9afb46 response_code = 200 uri = /network-bootstore-config
17552024-10-09T05:51:35.817ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43192
17562024-10-09T05:51:35.818ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 653f6089-0158-484c-84b9-bfd9e4d86b4f, kind: Service { id: d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59 }, name: Name("external-dns-d3b5ad95-aa59-4bb7-bebd-f1bd48e31d59"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 211, 149])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: cad5ed0d-3ea9-489d-93d4-7e3565b87eb6, kind: Service { id: 94415647-751f-4321-816f-7e4752b52496 }, name: Name("external-dns-94415647-751f-4321-816f-7e4752b52496"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 147, 7])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 08db3328-de2f-4bc0-aa29-88948e0b323e, kind: Service { id: a9346314-f5a8-4da4-8c60-48c5ab6c3dee }, name: Name("nexus-a9346314-f5a8-4da4-8c60-48c5ab6c3dee"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 29])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6296d238-bc00-44df-a0e2-f8ef31c024da, kind: Service { id: bcd392d4-4bfb-4d15-99aa-1265e84bc8dc }, name: Name("nexus-bcd392d4-4bfb-4d15-99aa-1265e84bc8dc"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 191, 204])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 7f2385eb-ba1c-4d5f-9e21-61e91356244e, kind: Service { id: 9a7d69b2-b5cb-4929-9d3e-5c8730f41abc }, name: Name("nexus-9a7d69b2-b5cb-4929-9d3e-5c8730f41abc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 96])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17572024-10-09T05:51:35.818ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17582024-10-09T05:51:35.818ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17592024-10-09T05:51:35.818ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17602024-10-09T05:51:35.818ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
17612024-10-09T05:51:35.818ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17622024-10-09T05:51:35.818ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17632024-10-09T05:51:35.818ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 428 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43192 req_id = 42e6a545-d5cb-4f5e-9b72-babbef15c534 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17642024-10-09T05:51:35.845ZINFOSledAgent (PortManager): skipping delta compute for subnet file = illumos-utils/src/opte/port_manager.rs:469 new_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 4 } old_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 4 } subnet = RouterId { vni: Vni(100), kind: System }
17652024-10-09T05:51:35.845ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 404 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38107 req_id = 14c21685-9b85-4763-9cda-b1a1a543f7da response_code = 204 uri = /vpc-routes
17662024-10-09T05:51:36.157ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60928
17672024-10-09T05:51:36.158ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60928 req_id = 9282df90-edcd-4d33-9b6e-8c8399038ac7 response_code = 200 uri = /v2p
17682024-10-09T05:51:36.217ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37287
17692024-10-09T05:51:36.217ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37287 req_id = 2e8f61c2-6bb1-4e2f-b112-0a0b88870abb response_code = 200 uri = /vpc-routes
17702024-10-09T05:51:36.329ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
17712024-10-09T05:51:36.329ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37287 req_id = 65d2f912-c71a-4df8-8821-c5c03cbdb12e response_code = 204 uri = /eip-gateways
17722024-10-09T05:51:36.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 253 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37287 req_id = c9bc804c-b063-4f3e-8ad2-146df3e43665 response_code = 204 uri = /vpc-routes
17732024-10-09T05:51:36.423ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33796
17742024-10-09T05:51:36.423ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
17752024-10-09T05:51:36.459ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
17762024-10-09T05:51:36.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88866 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33796 req_id = b5531c39-1b51-44e6-85d1-8fab82c2e7ef response_code = 204 uri = /switch-ports
17772024-10-09T05:51:36.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33796 req_id = 58c9a180-efd4-4443-b636-d1855ac4acd6 response_code = 200 uri = /network-bootstore-config
17782024-10-09T05:51:42.008ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51601
17792024-10-09T05:51:42.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51601 req_id = 137f9250-4787-4fd1-83d4-290229f5f9e2 response_code = 200 uri = /vpc-routes
17802024-10-09T05:51:42.028ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
17812024-10-09T05:51:42.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51601 req_id = fc262fee-cbb7-43f6-8964-e75e8b5e466f response_code = 204 uri = /eip-gateways
17822024-10-09T05:51:42.037ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51601 req_id = 100bb4db-d6e2-4b42-af95-566b5dc63d6a response_code = 204 uri = /vpc-routes
17832024-10-09T05:51:43.807ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55406
17842024-10-09T05:51:43.807ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55406 req_id = 6bf7c916-dc83-4f32-8a4d-9cf93bf8e2ff response_code = 200 uri = /baseboard
17852024-10-09T05:52:05.104ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36951
17862024-10-09T05:52:05.105ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36951 req_id = 44679eb4-db30-4f24-b83e-26dffe7cee20 response_code = 200 uri = /vpc-routes
17872024-10-09T05:52:05.169ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
17882024-10-09T05:52:05.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36951 req_id = 54044d43-d198-4890-aa7b-ef8648bf6942 response_code = 204 uri = /eip-gateways
17892024-10-09T05:52:05.182ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 238 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36951 req_id = a0c71605-5ed3-4ba8-aebf-f8fca00b584c response_code = 204 uri = /vpc-routes
17902024-10-09T05:52:05.220ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:33187 remote_addr = [fd00:1122:3344:101::d]:52907 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d
17912024-10-09T05:52:05.247ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 26120 local_addr = [fd00:1122:3344:101::1]:33187 method = GET remote_addr = [fd00:1122:3344:101::d]:52907 req_id = 23f73e15-78d3-4482-8e9f-c2784a19d62a response_code = 200 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d uri = /8fa2b36a-fea8-4835-8541-6d298f48930d
17922024-10-09T05:52:05.342ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54343
17932024-10-09T05:52:05.343ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54343 req_id = f8705561-5658-4833-a12b-9fd7532490a8 response_code = 200 uri = /v2p
17942024-10-09T05:52:05.364ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40889
17952024-10-09T05:52:05.364ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40889 req_id = 0c73cdb8-f605-44ec-8802-e16e938f3572 response_code = 200 uri = /vpc-routes
17962024-10-09T05:52:05.416ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
17972024-10-09T05:52:05.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 877 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40889 req_id = 1dc5b4d9-8dac-4862-bed1-97445c34afb5 response_code = 204 uri = /eip-gateways
17982024-10-09T05:52:05.462ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35244
17992024-10-09T05:52:05.463ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35244 req_id = a86cdbfe-1053-47c9-ac1d-bc8bce7d5c9b response_code = 200 uri = /v2p
18002024-10-09T05:52:05.515ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43984
18012024-10-09T05:52:05.516ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18022024-10-09T05:52:05.556ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18032024-10-09T05:52:05.612ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96895 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43984 req_id = 2ee0a318-694b-4c4d-b5da-4fbf12b88e2a response_code = 204 uri = /switch-ports
18042024-10-09T05:52:05.613ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 269 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43984 req_id = 758ca077-6f0e-4726-bfef-ded516be79f4 response_code = 200 uri = /network-bootstore-config
18052024-10-09T05:52:05.613ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 437 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40889 req_id = 8f72a9f5-ad0b-47b4-a23f-4a0cf76c1209 response_code = 204 uri = /vpc-routes
18062024-10-09T05:52:05.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40050
18072024-10-09T05:52:05.692ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18082024-10-09T05:52:05.730ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18092024-10-09T05:52:05.789ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96663 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40050 req_id = de12f4fe-c7de-41c1-ad68-9db3af9a4ceb response_code = 204 uri = /switch-ports
18102024-10-09T05:52:05.790ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40050 req_id = 4292ade3-b9a3-4e84-8e2e-8094410735e9 response_code = 200 uri = /network-bootstore-config
18112024-10-09T05:52:06.025ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44367
18122024-10-09T05:52:06.025ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44367 req_id = 0a2eeaae-54ac-489f-8582-99e68156564d response_code = 200 uri = /vpc-routes
18132024-10-09T05:52:06.082ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
18142024-10-09T05:52:06.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44367 req_id = d231af66-3102-4b34-9aa8-07cf21df1d24 response_code = 204 uri = /eip-gateways
18152024-10-09T05:52:06.092ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 401 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44367 req_id = 73f3e30f-3e75-4fd6-a644-611303eb858b response_code = 204 uri = /vpc-routes
18162024-10-09T05:52:06.242ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42325
18172024-10-09T05:52:06.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42325 req_id = 400cb6bf-7455-4891-81d2-5c0a5b21ec60 response_code = 200 uri = /v2p
18182024-10-09T05:52:06.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41961
18192024-10-09T05:52:06.311ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18202024-10-09T05:52:06.352ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18212024-10-09T05:52:06.412ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102037 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:41961 req_id = 6238bb15-ccac-47fb-be27-e56f9334d14f response_code = 204 uri = /switch-ports
18222024-10-09T05:52:06.413ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41961 req_id = 290dd45b-63cf-4a4d-a1c5-737b458501be response_code = 200 uri = /network-bootstore-config
18232024-10-09T05:52:13.840ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56640
18242024-10-09T05:52:13.841ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 639 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56640 req_id = 618cac25-bdd3-47ae-a30b-b45f752ceb6b response_code = 200 uri = /baseboard
18252024-10-09T05:52:34.930ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40055
18262024-10-09T05:52:34.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40055 req_id = 651b1132-7dff-499b-abe0-bf03f0de1d22 response_code = 200 uri = /vpc-routes
18272024-10-09T05:52:34.944ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61652
18282024-10-09T05:52:34.944ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61652 req_id = 97e34180-15ff-4041-bee7-524da51e27c0 response_code = 200 uri = /vpc-routes
18292024-10-09T05:52:34.951ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
18302024-10-09T05:52:34.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1195 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40055 req_id = 647d9c0d-606f-4e61-b5e6-4fe8f1ad3572 response_code = 204 uri = /eip-gateways
18312024-10-09T05:52:34.963ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 264 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40055 req_id = f11db736-e39a-4b1e-b373-6b466124f88e response_code = 204 uri = /vpc-routes
18322024-10-09T05:52:34.976ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
18332024-10-09T05:52:34.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61652 req_id = 7d785669-9652-4610-8ea7-ba7bed8c154b response_code = 204 uri = /eip-gateways
18342024-10-09T05:52:34.987ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 273 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61652 req_id = 6f83a806-14e5-4dc4-b013-60a60ec15427 response_code = 204 uri = /vpc-routes
18352024-10-09T05:52:35.172ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40005
18362024-10-09T05:52:35.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40005 req_id = 9e173c93-5fda-464b-8f95-86a3c6821097 response_code = 200 uri = /v2p
18372024-10-09T05:52:35.208ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58449
18382024-10-09T05:52:35.209ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 704 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58449 req_id = 8af24ddb-e1f4-42da-9148-f5b544b7790a response_code = 200 uri = /v2p
18392024-10-09T05:52:35.227ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6605 local_addr = [fd00:1122:3344:101::1]:33187 method = GET remote_addr = [fd00:1122:3344:101::d]:52907 req_id = 87940f31-3a34-4e4b-8362-c26e677dff0d response_code = 200 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d uri = /8fa2b36a-fea8-4835-8541-6d298f48930d
18402024-10-09T05:52:35.342ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51163
18412024-10-09T05:52:35.342ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18422024-10-09T05:52:35.379ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18432024-10-09T05:52:35.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96412 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:51163 req_id = 66ba04ea-f748-4b0a-b9ea-474c0aeaed64 response_code = 204 uri = /switch-ports
18442024-10-09T05:52:35.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51163 req_id = d8ccc663-a805-47e1-a8dc-74bed07a55e9 response_code = 200 uri = /network-bootstore-config
18452024-10-09T05:52:35.485ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56997
18462024-10-09T05:52:35.485ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18472024-10-09T05:52:35.525ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18482024-10-09T05:52:35.580ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94860 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:56997 req_id = 2338d6d9-cee2-4525-beb8-f3614cbe6f1b response_code = 204 uri = /switch-ports
18492024-10-09T05:52:35.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56997 req_id = cdae050e-031f-4fe2-9812-37ae193f836a response_code = 200 uri = /network-bootstore-config
18502024-10-09T05:52:36.098ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60257
18512024-10-09T05:52:36.098ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60257 req_id = 81337b3b-9261-4353-bc1b-b1d423aae6eb response_code = 200 uri = /vpc-routes
18522024-10-09T05:52:36.147ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
18532024-10-09T05:52:36.147ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 843 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60257 req_id = c0617d85-a20e-4c4a-a498-a9e8703560f3 response_code = 204 uri = /eip-gateways
18542024-10-09T05:52:36.160ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50919
18552024-10-09T05:52:36.160ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18562024-10-09T05:52:36.202ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18572024-10-09T05:52:36.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103620 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50919 req_id = ed2e19e0-afd5-4ffb-a6ff-e969746d906e response_code = 204 uri = /switch-ports
18582024-10-09T05:52:36.264ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59067
18592024-10-09T05:52:36.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50919 req_id = bcb4e82d-6a3e-4674-86fb-8d0c7ca87218 response_code = 200 uri = /network-bootstore-config
18602024-10-09T05:52:36.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 845 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59067 req_id = 6f7bf55d-725f-474c-8f26-1be6a3ef6f9c response_code = 200 uri = /v2p
18612024-10-09T05:52:36.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1143 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60257 req_id = c6ad395d-104f-489c-b0f5-d999b935ceee response_code = 204 uri = /vpc-routes
18622024-10-09T05:52:43.881ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65450
18632024-10-09T05:52:43.881ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 74 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65450 req_id = 21daeb62-a3aa-4ab0-ae7f-10af4cc49168 response_code = 200 uri = /baseboard
18642024-10-09T05:53:05.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40949
18652024-10-09T05:53:05.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40949 req_id = bf9bd8fe-bee8-4633-b2fd-fe52f8727ae9 response_code = 200 uri = /vpc-routes
18662024-10-09T05:53:05.231ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8676 local_addr = [fd00:1122:3344:101::1]:33187 method = GET remote_addr = [fd00:1122:3344:101::d]:52907 req_id = c01b9569-8f0f-47f5-8905-91721749d877 response_code = 200 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d uri = /8fa2b36a-fea8-4835-8541-6d298f48930d
18672024-10-09T05:53:05.245ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48511
18682024-10-09T05:53:05.245ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48511 req_id = 0ec13f1c-9931-4fe5-886e-8b7ac8aafd28 response_code = 200 uri = /v2p
18692024-10-09T05:53:05.313ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
18702024-10-09T05:53:05.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40949 req_id = b649737e-a27e-49a9-ad2d-9b17f0715b21 response_code = 204 uri = /eip-gateways
18712024-10-09T05:53:05.323ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40949 req_id = 1b620dbc-6158-4ac1-a236-fc8dd8dd7102 response_code = 204 uri = /vpc-routes
18722024-10-09T05:53:05.365ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34739
18732024-10-09T05:53:05.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34739 req_id = 091009b9-fd4b-4a3b-b232-d4dd2ad7cf12 response_code = 200 uri = /v2p
18742024-10-09T05:53:05.367ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62168
18752024-10-09T05:53:05.368ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 914 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62168 req_id = b8ecbccd-e9c2-480e-9416-e18b7e4d04c9 response_code = 200 uri = /vpc-routes
18762024-10-09T05:53:05.384ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
18772024-10-09T05:53:05.384ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 32 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62168 req_id = 4dd46936-9e41-4e0b-90e7-5809f51d802f response_code = 204 uri = /eip-gateways
18782024-10-09T05:53:05.395ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62168 req_id = b256da29-b3a5-46ff-b815-96863d1ace65 response_code = 204 uri = /vpc-routes
18792024-10-09T05:53:05.438ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52118
18802024-10-09T05:53:05.438ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18812024-10-09T05:53:05.474ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18822024-10-09T05:53:05.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91425 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52118 req_id = c7b1068b-7f50-45b0-b175-6c24106b4477 response_code = 204 uri = /switch-ports
18832024-10-09T05:53:05.530ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 287 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52118 req_id = 895a1fe3-a8d4-45d8-80e4-ab67368ad4ff response_code = 200 uri = /network-bootstore-config
18842024-10-09T05:53:05.630ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63763
18852024-10-09T05:53:05.631ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18862024-10-09T05:53:05.668ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18872024-10-09T05:53:05.721ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90924 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63763 req_id = 7f2f48d9-b26b-44d2-a6bc-d82ff03ed7db response_code = 204 uri = /switch-ports
18882024-10-09T05:53:05.722ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63763 req_id = 9ae94828-465f-486d-8b6c-da70bd836c3b response_code = 200 uri = /network-bootstore-config
18892024-10-09T05:53:06.215ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47861
18902024-10-09T05:53:06.215ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47861 req_id = ae53cce1-3fd7-4120-ad71-65536250efb6 response_code = 200 uri = /vpc-routes
18912024-10-09T05:53:06.258ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
18922024-10-09T05:53:06.258ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47861 req_id = c6185677-ee23-4676-bd78-ba9f81dbbd28 response_code = 204 uri = /eip-gateways
18932024-10-09T05:53:06.317ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63522
18942024-10-09T05:53:06.318ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63522 req_id = e61e5838-11be-41b1-aeab-6595e14d1380 response_code = 200 uri = /v2p
18952024-10-09T05:53:06.324ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 230 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47861 req_id = 86075622-6748-432a-a9b2-ee700ee53110 response_code = 204 uri = /vpc-routes
18962024-10-09T05:53:06.424ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52860
18972024-10-09T05:53:06.424ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
18982024-10-09T05:53:06.463ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
18992024-10-09T05:53:06.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95519 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52860 req_id = a0cacee3-be98-479c-a02e-1a5ee3c19586 response_code = 204 uri = /switch-ports
19002024-10-09T05:53:06.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52860 req_id = 43f60f71-2394-42c2-b06f-27dbf595730d response_code = 200 uri = /network-bootstore-config
19012024-10-09T05:53:06.592ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
19022024-10-09T05:53:13.917ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33537
19032024-10-09T05:53:13.918ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 734 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33537 req_id = 2b9f43b2-0167-4bb6-885a-a662a7d5b870 response_code = 200 uri = /baseboard
19042024-10-09T05:53:35.006ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34352
19052024-10-09T05:53:35.006ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34352 req_id = 4967c462-e12e-4a9c-af43-043b700db863 response_code = 200 uri = /vpc-routes
19062024-10-09T05:53:35.044ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42521
19072024-10-09T05:53:35.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42521 req_id = 0edd94ab-068d-4476-9ce8-6ca0b6884027 response_code = 200 uri = /vpc-routes
19082024-10-09T05:53:35.057ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
19092024-10-09T05:53:35.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34352 req_id = e8d75963-158b-46b7-9f81-a605886e1720 response_code = 204 uri = /eip-gateways
19102024-10-09T05:53:35.069ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 228 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34352 req_id = 1fd47598-a519-4c75-b9b4-6d8bb1bdc308 response_code = 204 uri = /vpc-routes
19112024-10-09T05:53:35.099ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
19122024-10-09T05:53:35.099ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42521 req_id = d1490db8-937e-49d6-b616-36eaceb925b0 response_code = 204 uri = /eip-gateways
19132024-10-09T05:53:35.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47399
19142024-10-09T05:53:35.101ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47399 req_id = b6deac4e-b9ef-4bc1-8563-68f622215c7b response_code = 200 uri = /v2p
19152024-10-09T05:53:35.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 991 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42521 req_id = d45beec4-7a79-4bec-a484-7b2da661dd12 response_code = 204 uri = /vpc-routes
19162024-10-09T05:53:35.160ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44016
19172024-10-09T05:53:35.160ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
19182024-10-09T05:53:35.198ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
19192024-10-09T05:53:35.256ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96617 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44016 req_id = 060bc411-e615-4ae6-a8e0-1aa165aeb13f response_code = 204 uri = /switch-ports
19202024-10-09T05:53:35.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59432
19212024-10-09T05:53:35.257ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 323 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44016 req_id = c2100175-3af1-4025-9443-66377d07895e response_code = 200 uri = /network-bootstore-config
19222024-10-09T05:53:35.257ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 317 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59432 req_id = 0f38349a-2f00-40b5-a563-cd7debee2322 response_code = 200 uri = /v2p
19232024-10-09T05:53:35.265ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8057 local_addr = [fd00:1122:3344:101::1]:33187 method = GET remote_addr = [fd00:1122:3344:101::d]:52907 req_id = 9a5c0de1-e7a2-4bc4-9145-dcf6f81817d5 response_code = 200 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d uri = /8fa2b36a-fea8-4835-8541-6d298f48930d
19242024-10-09T05:53:35.348ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50787
19252024-10-09T05:53:35.348ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
19262024-10-09T05:53:35.384ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
19272024-10-09T05:53:35.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90060 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50787 req_id = 5ff74d8c-5711-4e84-9451-e0e4bf972e84 response_code = 204 uri = /switch-ports
19282024-10-09T05:53:35.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 565 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50787 req_id = 4af1b513-84c0-494d-bffa-0899bfbb1eca response_code = 200 uri = /network-bootstore-config
19292024-10-09T05:53:36.084ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39005
19302024-10-09T05:53:36.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39005 req_id = 33505154-e0b1-4b70-afb0-a1b09f9a0a96 response_code = 200 uri = /vpc-routes
19312024-10-09T05:53:36.133ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1184 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d values = ExternalIpGatewayMap { mappings: {} }
19322024-10-09T05:53:36.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39005 req_id = 6f281b3c-cc2d-41f5-95c6-206c42479547 response_code = 204 uri = /eip-gateways
19332024-10-09T05:53:36.146ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 387 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39005 req_id = 7093bd2d-7f16-4704-8aef-e2f5c8ae4d8c response_code = 204 uri = /vpc-routes
19342024-10-09T05:53:36.172ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40947
19352024-10-09T05:53:36.172ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4116
19362024-10-09T05:53:36.212ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4140
19372024-10-09T05:53:36.272ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99431 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40947 req_id = fe012072-9b49-4245-a4de-8138c24e3947 response_code = 204 uri = /switch-ports
19382024-10-09T05:53:36.272ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40124
19392024-10-09T05:53:36.272ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 245 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40947 req_id = ff9ce225-7773-49c9-886a-3de7745267a1 response_code = 200 uri = /network-bootstore-config
19402024-10-09T05:53:36.273ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 922 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40124 req_id = 4ac65041-8b0e-416a-84d7-363cebbd37f6 response_code = 200 uri = /v2p
19412024-10-09T05:53:44.199ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56036
19422024-10-09T05:53:44.199ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 503 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56036 req_id = 21a0a95f-6bd8-4836-b4ee-c7142e76b947 response_code = 200 uri = /baseboard
19432024-10-09T05:53:46.037ZERROSledAgent (ProbeManager): get target probe state: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ec98a841-1182-45ad-a505-92384a5460d4", "content-length": "124", "date": "Wed, 09 Oct 2024 05:53:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ec98a841-1182-45ad-a505-92384a5460d4" } file = sled-agent/src/probe_manager.rs:230 sled_id = 8fa2b36a-fea8-4835-8541-6d298f48930d