Max level shown:
1 [ Oct 22 00:30:34 Enabled. ]
2 [ Oct 22 00:30:34 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Oct 22 00:30:34 Method "start" exited with status 0. ]
4 [ Oct 22 00:30:34 Rereading configuration. ]
5 [ Oct 22 00:30:34 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-10-22T00:30:34.402ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-10-22T00:30:34.402ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-10-22T00:30:34.454ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-10-22T00:30:34.454ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-10-22T00:30:34.566ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-10-22T00:30:34.575ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-10-22T00:30:34.583ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-10-22T00:30:35.188ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-10-22T00:30:35.323ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-10-22T00:30:35.323ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-10-22T00:30:35.323ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-10-22T00:30:35.323ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
192024-10-22T00:30:35.323ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
202024-10-22T00:30:35.323ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-10-22T00:30:35.326ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-10-22T00:30:35.326ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-10-22T00:30:35.326ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-10-22T00:30:35.326ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-10-22T00:30:35.326ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-10-22T00:30:35.326ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-10-22T00:30:35.326ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-10-22T00:30:35.326ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-10-22T00:30:35.326ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-10-22T00:30:35.337ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-10-22T00:30:35.337ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool)
322024-10-22T00:30:35.435ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal }
332024-10-22T00:30:35.842ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-10-22T00:30:35.842ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-10-22T00:30:35.842ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-10-22T00:30:35.842ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-10-22T00:30:35.842ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-10-22T00:30:35.842ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-10-22T00:30:35.842ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-10-22T00:30:35.851ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412024-10-22T00:30:35.851ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 042a0bdb-3057-4098-8648-7fd777304d2c (zpool)
422024-10-22T00:30:35.939ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal }
432024-10-22T00:30:36.359ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-10-22T00:30:36.359ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-10-22T00:30:36.359ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
472024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
482024-10-22T00:30:36.359ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-10-22T00:30:36.359ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-10-22T00:30:36.359ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-10-22T00:30:36.359ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
522024-10-22T00:30:36.359ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-10-22T00:30:36.359ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-10-22T00:30:36.359ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
572024-10-22T00:30:36.359ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
582024-10-22T00:30:36.359ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
602024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-10-22T00:30:36.359ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-10-22T00:30:36.359ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-10-22T00:30:36.359ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-10-22T00:30:36.359ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-10-22T00:30:36.359ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-10-22T00:30:36.359ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-10-22T00:30:36.360ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-10-22T00:30:36.360ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-10-22T00:30:36.360ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-10-22T00:30:36.360ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-10-22T00:30:36.360ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-10-22T00:30:36.360ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
762024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
772024-10-22T00:30:36.360ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
782024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-10-22T00:30:36.360ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-10-22T00:30:36.360ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
822024-10-22T00:30:36.360ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
832024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
842024-10-22T00:30:36.360ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852024-10-22T00:30:36.360ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-10-22T00:30:36.360ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
882024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
892024-10-22T00:30:36.360ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
902024-10-22T00:30:36.360ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-10-22T00:30:36.360ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-10-22T00:30:36.360ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-10-22T00:30:36.360ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
942024-10-22T00:30:36.360ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
952024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-10-22T00:30:36.360ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-10-22T00:30:36.360ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-10-22T00:30:36.360ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-10-22T00:30:36.360ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1022024-10-22T00:30:36.360ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-10-22T00:30:36.360ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-10-22T00:30:36.360ZINFOSledAgent (bootstore): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-10-22T00:30:36.360ZINFOSledAgent (bootstore): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-10-22T00:30:36.360ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-10-22T00:30:36.360ZINFOSledAgent (bootstore): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-10-22T00:30:36.360ZINFOSledAgent (bootstore): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-10-22T00:30:36.360ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-10-22T00:30:36.360ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-10-22T00:30:36.360ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-10-22T00:30:36.370ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:742
1132024-10-22T00:30:36.370ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3961
1142024-10-22T00:30:36.370ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1587, tv_nsec: 64636622 } time_to_next_cleanup = 599.999924209s
1152024-10-22T00:30:36.370ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-10-22T00:30:36.370ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-10-22T00:30:36.372ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-10-22T00:30:36.373ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-10-22T00:30:36.375ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-10-22T00:30:36.375ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1073
1212024-10-22T00:30:36.375ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1222024-10-22T00:30:36.375ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-10-22T00:30:36.375ZINFOSledAgent (RSS): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-10-22T00:30:36.375ZINFOSledAgent (RSS): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-10-22T00:30:36.375ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1136
1262024-10-22T00:30:36.375ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-10-22T00:30:36.375ZINFOSledAgent (RSS): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-10-22T00:30:36.375ZINFOSledAgent (RSS): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-10-22T00:30:36.375ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1178
1302024-10-22T00:30:36.375ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-10-22T00:30:36.375ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-10-22T00:30:36.375ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-10-22T00:30:36.376ZINFOSledAgent (RSS): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-10-22T00:30:36.376ZINFOSledAgent (RSS): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-10-22T00:30:36.376ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 26fc8a25-8c7c-49c9-be24-a8461466dfe6, sleds: { [fdb0:18c0:4d0d:a02a::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: e50e0974-d68b-427b-915b-5bd9a655288a (sled), rack_id: 26fc8a25-8c7c-49c9-be24-a8461466dfe6, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-10-22T00:30:36.376ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1212
1372024-10-22T00:30:36.376ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-10-22T00:30:36.376ZINFOSledAgent (bootstore): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-10-22T00:30:36.376ZINFOSledAgent (bootstore): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-10-22T00:30:36.376ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-10-22T00:30:36.377ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: e50e0974-d68b-427b-915b-5bd9a655288a (sled), rack_id: 26fc8a25-8c7c-49c9-be24-a8461466dfe6, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-10-22T00:30:36.377ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4258
1432024-10-22T00:30:36.377ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4658
1442024-10-22T00:30:36.381ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-10-22T00:30:36.387ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-10-22T00:30:36.387ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-10-22T00:30:36.391ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-10-22T00:30:36.392ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-10-22T00:30:36.392ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:a02a::1]:58389
1502024-10-22T00:30:36.394ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: e50e0974-d68b-427b-915b-5bd9a655288a (sled), rack_id: 26fc8a25-8c7c-49c9-be24-a8461466dfe6, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1512024-10-22T00:30:36.394ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1522024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1532024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1542024-10-22T00:30:36.394ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1552024-10-22T00:30:36.394ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1562024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1572024-10-22T00:30:36.394ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1582024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-10-22T00:30:36.394ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-10-22T00:30:36.394ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-10-22T00:30:36.406ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-10-22T00:30:36.426ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-10-22T00:30:36.443ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1672024-10-22T00:30:36.474ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
1682024-10-22T00:30:36.474ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1692024-10-22T00:30:36.474ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1702024-10-22T00:30:36.474ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
1712024-10-22T00:30:36.476ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 63.46328ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "69384584-e948-49c7-b976-2f967b3f5443", "content-length": "124", "date": "Tue, 22 Oct 2024 00:30:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "69384584-e948-49c7-b976-2f967b3f5443" } file = clients/ddm-admin-client/src/lib.rs:177
1722024-10-22T00:30:36.487ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_20b2e8dd-3528-47c6-ab14-1bee850adf61/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-10-22T00:30:36.510ZINFOSledAgent: successfully created encrypted zvol: name="oxi_20b2e8dd-3528-47c6-ab14-1bee850adf61/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-10-22T00:30:36.511ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_20b2e8dd-3528-47c6-ab14-1bee850adf61/swap" file = sled-agent/src/swap_device.rs:96
1752024-10-22T00:30:36.516ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
1762024-10-22T00:30:36.516ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1772024-10-22T00:30:36.516ZINFOSledAgent (BackingFs): Ensuring dataset oxi_20b2e8dd-3528-47c6-ab14-1bee850adf61/backing/fmd file = sled-agent/src/backing_fs.rs:138
1782024-10-22T00:30:36.541ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-10-22T00:30:36.542ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 138.823076ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "96159510-230d-42d6-82b4-ec63bda15023", "content-length": "124", "date": "Tue, 22 Oct 2024 00:30:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "96159510-230d-42d6-82b4-ec63bda15023" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-10-22T00:30:36.602ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1812024-10-22T00:30:36.681ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-10-22T00:30:36.682ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 136.050982ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a0a612e6-2eb9-42a6-910d-cf04eb6bf3a4", "content-length": "124", "date": "Tue, 22 Oct 2024 00:30:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a0a612e6-2eb9-42a6-910d-cf04eb6bf3a4" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-10-22T00:30:36.820ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-10-22T00:30:36.820ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 120.318228ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "870e2f6b-5bed-4217-a9fc-61984745eebf", "content-length": "124", "date": "Tue, 22 Oct 2024 00:30:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "870e2f6b-5bed-4217-a9fc-61984745eebf" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-10-22T00:30:36.821ZINFOSledAgent (BackingFs): Mounting oxi_20b2e8dd-3528-47c6-ab14-1bee850adf61/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1862024-10-22T00:30:36.834ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1872024-10-22T00:30:36.834ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1882024-10-22T00:30:36.834ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1892024-10-22T00:30:36.834ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1902024-10-22T00:30:36.941ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-10-22T00:30:36.942ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 231.513548ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "93ccc930-1d4e-4cb7-a6e9-456c6774978f", "content-length": "124", "date": "Tue, 22 Oct 2024 00:30:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "93ccc930-1d4e-4cb7-a6e9-456c6774978f" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-10-22T00:30:37.174ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-10-22T00:30:37.175ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 225.117876ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "75923024-ae23-4e53-bc18-7f0696b56b78", "content-length": "124", "date": "Tue, 22 Oct 2024 00:30:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "75923024-ae23-4e53-bc18-7f0696b56b78" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-10-22T00:30:37.400ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-10-22T00:30:37.401ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 184.596828ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4104a8d8-0ade-47d5-baa4-71873705b368", "content-length": "124", "date": "Tue, 22 Oct 2024 00:30:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4104a8d8-0ade-47d5-baa4-71873705b368" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-10-22T00:30:37.562ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
1972024-10-22T00:30:37.567ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:54049 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
1982024-10-22T00:30:37.567ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
1992024-10-22T00:30:37.567ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:54049 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = e50e0974-d68b-427b-915b-5bd9a655288a sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2002024-10-22T00:30:37.568ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 302.790714ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2012024-10-22T00:30:37.569ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2022024-10-22T00:30:37.569ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2032024-10-22T00:30:37.569ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:926 underlay_address = fd00:1122:3344:101::1
2042024-10-22T00:30:37.569ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
2052024-10-22T00:30:37.586ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2062024-10-22T00:30:37.873ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 363.93335ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2072024-10-22T00:30:38.239ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.201150622s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2082024-10-22T00:30:39.442ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.98888273s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2092024-10-22T00:30:40.145ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2683
2102024-10-22T00:30:40.145ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2928
2112024-10-22T00:30:40.145ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2553
2122024-10-22T00:30:40.145ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2600
2132024-10-22T00:30:40.145ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:3009
2142024-10-22T00:30:40.145ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3054
2152024-10-22T00:30:40.147ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2853
2162024-10-22T00:30:40.148ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2594
2172024-10-22T00:30:40.148ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='fe42448e-4544-42ed-88ea-45a52d605c32'/> <propval type="astring" name="rack_id" value='26fc8a25-8c7c-49c9-be24-a8461466dfe6'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fef4:491b'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="rack_id" value='26fc8a25-8c7c-49c9-be24-a8461466dfe6'/> <propval type="astring" name="sled_id" value='e50e0974-d68b-427b-915b-5bd9a655288a'/> <propval type="astring" name="sled_model" value='i86pc'/> <propval type="astring" name="sled_serial" value='sock'/> <propval type="astring" name="sled_revision" value='0'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='e50e0974-d68b-427b-915b-5bd9a655288a'/> <propval type="astring" name="rack_uuid" value='26fc8a25-8c7c-49c9-be24-a8461466dfe6'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='e50e0974-d68b-427b-915b-5bd9a655288a'/> <propval type="astring" name="rack_uuid" value='26fc8a25-8c7c-49c9-be24-a8461466dfe6'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2182024-10-22T00:30:40.149ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2192024-10-22T00:30:40.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 64.379354ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-10-22T00:30:40.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 116.631452ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-10-22T00:30:40.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 228.569995ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-10-22T00:30:41.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 223.291703ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-10-22T00:30:41.432ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 3.402082212s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2242024-10-22T00:30:41.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 819.7458ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-10-22T00:30:42.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.274499452s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-10-22T00:30:43.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.452058076s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-10-22T00:30:44.836ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 9.645159299s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2282024-10-22T00:30:45.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.04759497s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2292024-10-22T00:30:47.077ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:12345
2302024-10-22T00:30:47.077ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2312024-10-22T00:30:47.077ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2322024-10-22T00:30:47.077ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2332024-10-22T00:30:47.077ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2342024-10-22T00:30:47.077ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2352024-10-22T00:30:47.077ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2362024-10-22T00:30:47.077ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3961
2372024-10-22T00:30:47.326ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:a02a::1]:58389
2382024-10-22T00:30:47.327ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2392024-10-22T00:30:47.327ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2402024-10-22T00:30:47.327ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-10-22T00:30:47.327ZINFOSledAgent (RSS): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2422024-10-22T00:30:47.327ZINFOSledAgent (RSS): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2432024-10-22T00:30:47.327ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-10-22T00:30:47.327ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-10-22T00:30:47.327ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-10-22T00:30:47.332ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4282 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2472024-10-22T00:30:47.332ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4298
2482024-10-22T00:30:47.332ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2492024-10-22T00:30:47.584ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12602934267546915325 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2502024-10-22T00:30:47.584ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4098285966382031446 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2512024-10-22T00:30:47.613ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53154
2522024-10-22T00:30:47.613ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2532024-10-22T00:30:47.613ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2542024-10-22T00:30:47.613ZINFOSledAgent (ServiceManager): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-zones.json file = common/src/ledger.rs:188
2552024-10-22T00:30:47.613ZINFOSledAgent (ServiceManager): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-zones.json file = common/src/ledger.rs:188
2562024-10-22T00:30:47.614ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 314 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53154 req_id = 4f2718fa-60df-4acd-991d-e3f646edd129 response_code = 200 uri = /inventory
2572024-10-22T00:30:47.643ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49037
2582024-10-22T00:30:47.643ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49037 req_id = c78d8752-b400-470f-be25-5e24a0fa3363 response_code = 200 uri = /sled-role
2592024-10-22T00:30:47.645ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2602024-10-22T00:30:47.645ZINFOSledAgent (RSS): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2612024-10-22T00:30:47.645ZINFOSledAgent (RSS): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2622024-10-22T00:30:47.645ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:882
2632024-10-22T00:30:47.675ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 8bd9a483-535a-4e7b-9ab1-8ac31df060a8, pool_id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: f66b0c75-706c-4006-bf36-f33374019817, pool_id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: dbe235e4-ebaf-4d8f-a1c9-b359a04f2a07, pool_id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 545915ca-d5ff-4e81-9f66-d6a4325c0d78, pool_id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: b0878cd1-90d7-48da-b4a2-e1e2c6c72a2d, pool_id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: de3164fd-78e9-4d3c-b327-149a4740db1c, pool_id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 6fccb824-9c2f-4408-b2e2-0cb20a6da0a2, pool_id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 9f0d1b8e-7b9c-486f-a31c-d56dff8302e0, pool_id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: aeb98aa9-9753-47ec-9039-ab27648f789a, pool_id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2642024-10-22T00:30:47.675ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42835
2652024-10-22T00:30:47.676ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:862 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2662024-10-22T00:30:47.676ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 8bd9a483-535a-4e7b-9ab1-8ac31df060a8, pool_id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: f66b0c75-706c-4006-bf36-f33374019817, pool_id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: dbe235e4-ebaf-4d8f-a1c9-b359a04f2a07, pool_id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 545915ca-d5ff-4e81-9f66-d6a4325c0d78, pool_id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: b0878cd1-90d7-48da-b4a2-e1e2c6c72a2d, pool_id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: de3164fd-78e9-4d3c-b327-149a4740db1c, pool_id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 6fccb824-9c2f-4408-b2e2-0cb20a6da0a2, pool_id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 9f0d1b8e-7b9c-486f-a31c-d56dff8302e0, pool_id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: aeb98aa9-9753-47ec-9039-ab27648f789a, pool_id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2672024-10-22T00:30:47.676ZINFOSledAgent (StorageManager): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2682024-10-22T00:30:47.676ZINFOSledAgent (StorageManager): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2692024-10-22T00:30:47.676ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2702024-10-22T00:30:47.676ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2712024-10-22T00:30:47.676ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2722024-10-22T00:30:47.676ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2732024-10-22T00:30:47.676ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2742024-10-22T00:30:47.828ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2752024-10-22T00:30:47.828ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool)
2762024-10-22T00:30:48.045ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2772024-10-22T00:30:48.082ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/e50e0974-d68b-427b-915b-5bd9a655288a) file = sled-agent/src/probe_manager.rs:230 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
2782024-10-22T00:30:48.142ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }
2792024-10-22T00:30:48.150ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2802024-10-22T00:30:48.150ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2812024-10-22T00:30:48.150ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2822024-10-22T00:30:48.150ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2832024-10-22T00:30:48.150ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2842024-10-22T00:30:48.151ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2852024-10-22T00:30:48.151ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2862024-10-22T00:30:48.184ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2872024-10-22T00:30:48.192ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_05815180-fa73-47c8-8793-9dad48a6a2a3/crypt/zone file = sled-storage/src/dataset.rs:280
2882024-10-22T00:30:48.359ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_05815180-fa73-47c8-8793-9dad48a6a2a3 file = sled-storage/src/dataset.rs:366
2892024-10-22T00:30:48.366ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2902024-10-22T00:30:48.366ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2912024-10-22T00:30:48.366ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2922024-10-22T00:30:48.366ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2932024-10-22T00:30:48.366ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2942024-10-22T00:30:48.376ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2952024-10-22T00:30:48.376ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 4912870c-99c9-4864-9312-2d393933ce6a (zpool)
2962024-10-22T00:30:48.466ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }
2972024-10-22T00:30:48.475ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2982024-10-22T00:30:48.475ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
2992024-10-22T00:30:48.475ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
3002024-10-22T00:30:48.475ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
3012024-10-22T00:30:48.475ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3022024-10-22T00:30:48.475ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3032024-10-22T00:30:48.475ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3042024-10-22T00:30:48.508ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3052024-10-22T00:30:48.516ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4912870c-99c9-4864-9312-2d393933ce6a/crypt/zone file = sled-storage/src/dataset.rs:280
3062024-10-22T00:30:48.689ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4912870c-99c9-4864-9312-2d393933ce6a file = sled-storage/src/dataset.rs:366
3072024-10-22T00:30:48.697ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3082024-10-22T00:30:48.697ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3092024-10-22T00:30:48.697ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3102024-10-22T00:30:48.697ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3112024-10-22T00:30:48.697ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3122024-10-22T00:30:48.707ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3132024-10-22T00:30:48.707ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool)
3142024-10-22T00:30:48.795ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }
3152024-10-22T00:30:48.802ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3162024-10-22T00:30:48.803ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3172024-10-22T00:30:48.803ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3182024-10-22T00:30:48.803ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3192024-10-22T00:30:48.803ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3202024-10-22T00:30:48.803ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3212024-10-22T00:30:48.803ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3222024-10-22T00:30:48.837ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3232024-10-22T00:30:48.845ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f643b80c-1fc2-4be9-93a9-fa7fd37e716e/crypt/zone file = sled-storage/src/dataset.rs:280
3242024-10-22T00:30:49.012ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f643b80c-1fc2-4be9-93a9-fa7fd37e716e file = sled-storage/src/dataset.rs:366
3252024-10-22T00:30:49.020ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3262024-10-22T00:30:49.020ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3272024-10-22T00:30:49.020ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3282024-10-22T00:30:49.020ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3292024-10-22T00:30:49.020ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3302024-10-22T00:30:49.030ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3312024-10-22T00:30:49.030ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool)
3322024-10-22T00:30:49.123ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }
3332024-10-22T00:30:49.131ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3342024-10-22T00:30:49.131ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3352024-10-22T00:30:49.132ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3362024-10-22T00:30:49.132ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3372024-10-22T00:30:49.132ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3382024-10-22T00:30:49.132ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3392024-10-22T00:30:49.132ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3402024-10-22T00:30:49.135ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/e50e0974-d68b-427b-915b-5bd9a655288a) file = sled-agent/src/probe_manager.rs:230 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
3412024-10-22T00:30:49.167ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3422024-10-22T00:30:49.174ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f3053f19-a930-43cc-8978-6dcb10f47d61/crypt/zone file = sled-storage/src/dataset.rs:280
3432024-10-22T00:30:49.345ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f3053f19-a930-43cc-8978-6dcb10f47d61 file = sled-storage/src/dataset.rs:366
3442024-10-22T00:30:49.353ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3452024-10-22T00:30:49.354ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3462024-10-22T00:30:49.354ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3472024-10-22T00:30:49.354ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3482024-10-22T00:30:49.354ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3492024-10-22T00:30:49.363ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3502024-10-22T00:30:49.363ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool)
3512024-10-22T00:30:49.456ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }
3522024-10-22T00:30:49.463ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3532024-10-22T00:30:49.463ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3542024-10-22T00:30:49.463ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3552024-10-22T00:30:49.463ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3562024-10-22T00:30:49.463ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3572024-10-22T00:30:49.463ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3582024-10-22T00:30:49.463ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3592024-10-22T00:30:49.497ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3602024-10-22T00:30:49.504ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_d912c6f5-c454-43da-ae16-2d232f5f0eb5/crypt/zone file = sled-storage/src/dataset.rs:280
3612024-10-22T00:30:49.672ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_d912c6f5-c454-43da-ae16-2d232f5f0eb5 file = sled-storage/src/dataset.rs:366
3622024-10-22T00:30:49.680ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3632024-10-22T00:30:49.680ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3642024-10-22T00:30:49.680ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3652024-10-22T00:30:49.680ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3662024-10-22T00:30:49.680ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3672024-10-22T00:30:49.690ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3682024-10-22T00:30:49.690ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = f72e2a71-8f15-4409-923c-af354b7632aa (zpool)
3692024-10-22T00:30:49.785ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External }
3702024-10-22T00:30:49.793ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3712024-10-22T00:30:49.793ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3722024-10-22T00:30:49.793ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3732024-10-22T00:30:49.793ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3742024-10-22T00:30:49.793ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3752024-10-22T00:30:49.793ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3762024-10-22T00:30:49.793ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3772024-10-22T00:30:49.827ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3782024-10-22T00:30:49.835ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f72e2a71-8f15-4409-923c-af354b7632aa/crypt/zone file = sled-storage/src/dataset.rs:280
3792024-10-22T00:30:50.005ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f72e2a71-8f15-4409-923c-af354b7632aa file = sled-storage/src/dataset.rs:366
3802024-10-22T00:30:50.012ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3812024-10-22T00:30:50.012ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3822024-10-22T00:30:50.012ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3832024-10-22T00:30:50.012ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3842024-10-22T00:30:50.012ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3852024-10-22T00:30:50.022ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3862024-10-22T00:30:50.022ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool)
3872024-10-22T00:30:50.122ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }
3882024-10-22T00:30:50.129ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3892024-10-22T00:30:50.129ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3902024-10-22T00:30:50.129ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3912024-10-22T00:30:50.129ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3922024-10-22T00:30:50.129ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3932024-10-22T00:30:50.130ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3942024-10-22T00:30:50.130ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3952024-10-22T00:30:50.140ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/e50e0974-d68b-427b-915b-5bd9a655288a) file = sled-agent/src/probe_manager.rs:230 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
3962024-10-22T00:30:50.165ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3972024-10-22T00:30:50.172ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4124bd0f-6d92-452d-b8ec-890efde7c175/crypt/zone file = sled-storage/src/dataset.rs:280
3982024-10-22T00:30:50.342ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4124bd0f-6d92-452d-b8ec-890efde7c175 file = sled-storage/src/dataset.rs:366
3992024-10-22T00:30:50.350ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
4002024-10-22T00:30:50.350ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
4012024-10-22T00:30:50.350ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4022024-10-22T00:30:50.350ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4032024-10-22T00:30:50.350ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4042024-10-22T00:30:50.359ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4052024-10-22T00:30:50.359ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool)
4062024-10-22T00:30:50.457ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool), kind: External }
4072024-10-22T00:30:50.466ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4082024-10-22T00:30:50.466ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4092024-10-22T00:30:50.466ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4102024-10-22T00:30:50.466ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4112024-10-22T00:30:50.466ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4122024-10-22T00:30:50.466ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4132024-10-22T00:30:50.466ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4142024-10-22T00:30:50.504ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4152024-10-22T00:30:50.513ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6bea9bba-836b-44a8-977b-0b50b3178081/crypt/zone file = sled-storage/src/dataset.rs:280
4162024-10-22T00:30:50.689ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6bea9bba-836b-44a8-977b-0b50b3178081 file = sled-storage/src/dataset.rs:366
4172024-10-22T00:30:50.697ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4182024-10-22T00:30:50.697ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4192024-10-22T00:30:50.697ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4202024-10-22T00:30:50.697ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4212024-10-22T00:30:50.697ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4222024-10-22T00:30:50.707ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4232024-10-22T00:30:50.707ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool)
4242024-10-22T00:30:50.805ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool), kind: External }
4252024-10-22T00:30:50.813ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4262024-10-22T00:30:50.813ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4272024-10-22T00:30:50.813ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4282024-10-22T00:30:50.813ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4292024-10-22T00:30:50.813ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4302024-10-22T00:30:50.814ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4312024-10-22T00:30:50.814ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4322024-10-22T00:30:50.849ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4332024-10-22T00:30:50.856ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c2c09b26-0ffd-409b-9a74-0a32dc4478f3/crypt/zone file = sled-storage/src/dataset.rs:280
4342024-10-22T00:30:51.022ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c2c09b26-0ffd-409b-9a74-0a32dc4478f3 file = sled-storage/src/dataset.rs:366
4352024-10-22T00:30:51.029ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4362024-10-22T00:30:51.029ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4372024-10-22T00:30:51.029ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4382024-10-22T00:30:51.029ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 20b2e8dd-3528-47c6-ab14-1bee850adf61 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 042a0bdb-3057-4098-8648-7fd777304d2c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4392024-10-22T00:30:51.029ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4402024-10-22T00:30:51.029ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4412024-10-22T00:30:51.029ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4422024-10-22T00:30:51.030ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:866 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4432024-10-22T00:30:51.030ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4442024-10-22T00:30:51.030ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:892 generation = Generation(1) sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4452024-10-22T00:30:51.030ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:897 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4462024-10-22T00:30:51.030ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:902 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4472024-10-22T00:30:51.030ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4482024-10-22T00:30:51.030ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:908 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4492024-10-22T00:30:51.030ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:798 gen = Generation(1)
4502024-10-22T00:30:51.030ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:913 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4512024-10-22T00:30:51.030ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3354164 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42835 req_id = 65fd8404-fff9-450e-8c02-35d3bff60d58 response_code = 200 uri = /omicron-physical-disks
4522024-10-22T00:30:51.061ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4532024-10-22T00:30:51.061ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57337
4542024-10-22T00:30:51.061ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4552024-10-22T00:30:51.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4562024-10-22T00:30:51.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4572024-10-22T00:30:51.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }
4582024-10-22T00:30:51.142ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/e50e0974-d68b-427b-915b-5bd9a655288a) file = sled-agent/src/probe_manager.rs:230 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4592024-10-22T00:30:51.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4602024-10-22T00:30:51.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }
4612024-10-22T00:30:51.209ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4622024-10-22T00:30:51.209ZINFOSledAgent (ServiceManager): No ledger in /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-zones.json file = common/src/ledger.rs:188
4632024-10-22T00:30:51.209ZINFOSledAgent (ServiceManager): No ledger in /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-zones.json file = common/src/ledger.rs:188
4642024-10-22T00:30:51.209ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4652024-10-22T00:30:51.252ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4662024-10-22T00:30:51.252ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4672024-10-22T00:30:51.252ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4682024-10-22T00:30:51.305ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d file = illumos-utils/src/zone.rs:316
4692024-10-22T00:30:51.309ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad file = illumos-utils/src/zone.rs:316
4702024-10-22T00:30:51.313ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7 file = illumos-utils/src/zone.rs:316
4712024-10-22T00:30:51.325ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d file = illumos-utils/src/zone.rs:354
4722024-10-22T00:30:51.329ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad file = illumos-utils/src/zone.rs:354
4732024-10-22T00:30:51.333ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7 file = illumos-utils/src/zone.rs:354
4742024-10-22T00:30:52.145ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/e50e0974-d68b-427b-915b-5bd9a655288a) file = sled-agent/src/probe_manager.rs:230 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
4752024-10-22T00:30:52.862ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4762024-10-22T00:30:52.862ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4772024-10-22T00:30:53.338ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4782024-10-22T00:30:53.815ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4792024-10-22T00:30:53.815ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4802024-10-22T00:30:53.815ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
4812024-10-22T00:30:53.819ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4822024-10-22T00:30:53.819ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
4832024-10-22T00:30:53.824ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
4842024-10-22T00:30:53.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 59.623304ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
4852024-10-22T00:30:53.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 65.095041ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
4862024-10-22T00:30:53.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 25.013234ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
4872024-10-22T00:30:53.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 132.325984ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
4882024-10-22T00:30:54.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 80.499138ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
4892024-10-22T00:30:54.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 88.329526ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
4902024-10-22T00:30:54.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 243.276059ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
4912024-10-22T00:30:54.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 278.180418ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
4922024-10-22T00:30:54.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 239.550385ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
4932024-10-22T00:30:54.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 229.554555ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
4942024-10-22T00:30:54.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 331.789437ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
4952024-10-22T00:30:54.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 250.346727ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
4962024-10-22T00:30:54.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 1.179820165s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
4972024-10-22T00:30:54.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 528.422063ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
4982024-10-22T00:30:55.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 460.208679ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
4992024-10-22T00:30:55.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 906.001137ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
5002024-10-22T00:30:55.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 1.197378258s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
5012024-10-22T00:30:56.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 1.344165566s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
5022024-10-22T00:30:56.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 1.242193272s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
5032024-10-22T00:30:56.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 848.653922ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
5042024-10-22T00:30:56.853ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4306
5052024-10-22T00:30:56.853ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4316 gateway = fd00:1122:3344:101::1
5062024-10-22T00:30:56.858ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4395
5072024-10-22T00:30:56.995ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4461
5082024-10-22T00:30:56.995ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4488
5092024-10-22T00:30:57.035ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4501
5102024-10-22T00:30:57.035ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4347
5112024-10-22T00:30:57.081ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8308348012536343278 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
5122024-10-22T00:30:57.082ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2561833297137706320 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
5132024-10-22T00:30:57.087ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4389
5142024-10-22T00:30:57.087ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4468 rack_subnet = fd00:1122:3344::
5152024-10-22T00:30:57.107ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4479
5162024-10-22T00:30:57.107ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4521
5172024-10-22T00:30:57.175ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4553
5182024-10-22T00:30:57.175ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4559
5192024-10-22T00:30:57.263ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4600
5202024-10-22T00:30:57.263ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
5212024-10-22T00:30:57.296ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 60.205609ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5222024-10-22T00:30:57.357ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 133.606089ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5232024-10-22T00:30:57.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 1.213988448s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
5242024-10-22T00:30:57.492ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 181.399854ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5252024-10-22T00:30:57.587ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12602934267546915325 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
5262024-10-22T00:30:57.587ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4098285966382031446 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
5272024-10-22T00:30:57.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 832.062001ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
5282024-10-22T00:30:57.674ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 514.719614ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5292024-10-22T00:30:57.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 738.713894ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
5302024-10-22T00:30:58.191ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:418
5312024-10-22T00:30:58.223ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:686
5322024-10-22T00:30:58.223ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:724
5332024-10-22T00:30:58.223ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:727 uptime = 1
5342024-10-22T00:30:58.223ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:440
5352024-10-22T00:30:58.259ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
5362024-10-22T00:30:58.312ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5372024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5382024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5392024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5402024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5412024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5422024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5432024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5442024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5452024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5462024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5472024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5482024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5492024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5502024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5512024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5522024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5532024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5542024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5552024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5562024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5572024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5582024-10-22T00:30:58.312ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5592024-10-22T00:30:58.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7") failed: Property not found. retry in 1.344625978s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
5602024-10-22T00:30:58.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d") failed: Property not found. retry in 1.020115936s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
5612024-10-22T00:30:58.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad") failed: Property not found. retry in 1.23440813s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
5622024-10-22T00:30:59.909ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_internal_dns_3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d
5632024-10-22T00:30:59.909ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_internal_dns_927912b9-9e81-49b4-8c6d-401ffc3d66ad
5642024-10-22T00:30:59.909ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_internal_dns_42eaca0b-e1f8-45e1-96a4-db2b312ecbf7
5652024-10-22T00:30:59.909ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5662024-10-22T00:30:59.909ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5672024-10-22T00:30:59.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8848695 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:57337 req_id = 606e13da-49d3-461d-b1b8-e5043f9287f0 response_code = 204 uri = /omicron-zones
5682024-10-22T00:30:59.911ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5692024-10-22T00:30:59.959ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5702024-10-22T00:31:00.004ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5712024-10-22T00:31:00.052ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5722024-10-22T00:31:00.052ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5732024-10-22T00:31:00.085ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5742024-10-22T00:31:00.086ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5752024-10-22T00:31:00.086ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5762024-10-22T00:31:00.115ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 7db72912-69be-4df2-ab6e-2dd45407474e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 7934b6c4-cfc3-4e03-b8b0-2ad4112eaa3b, kind: Service { id: 7db72912-69be-4df2-ab6e-2dd45407474e }, name: Name("ntp-7db72912-69be-4df2-ab6e-2dd45407474e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 146, 28])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5772024-10-22T00:31:00.115ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62070
5782024-10-22T00:31:00.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5792024-10-22T00:31:00.116ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5802024-10-22T00:31:00.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5812024-10-22T00:31:00.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }
5822024-10-22T00:31:00.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5832024-10-22T00:31:00.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }
5842024-10-22T00:31:00.173ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5852024-10-22T00:31:00.173ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-zones.json file = common/src/ledger.rs:177
5862024-10-22T00:31:00.174ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-zones.json file = common/src/ledger.rs:177
5872024-10-22T00:31:00.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5882024-10-22T00:31:00.185ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5892024-10-22T00:31:00.218ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5902024-10-22T00:31:00.219ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5912024-10-22T00:31:00.219ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5922024-10-22T00:31:00.248ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 146, 28]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5932024-10-22T00:31:00.248ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1270 zone_type = boundary_ntp
5942024-10-22T00:31:00.248ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
5952024-10-22T00:31:00.248ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
5962024-10-22T00:31:00.249ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
5972024-10-22T00:31:00.250ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5982024-10-22T00:31:00.273ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e file = illumos-utils/src/zone.rs:316
5992024-10-22T00:31:00.291ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e file = illumos-utils/src/zone.rs:354
6002024-10-22T00:31:00.910ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6012024-10-22T00:31:00.910ZINFOSledAgent (ServiceManager): Booting oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6022024-10-22T00:31:01.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 64.514097ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6032024-10-22T00:31:01.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 69.8168ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6042024-10-22T00:31:01.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 236.918785ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6052024-10-22T00:31:01.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 542.886789ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6062024-10-22T00:31:02.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 881.284449ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6072024-10-22T00:31:02.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 1.274796427s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6082024-10-22T00:31:04.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 901.2062ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6092024-10-22T00:31:05.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e") failed: Property not found. retry in 892.583761ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6102024-10-22T00:31:06.016ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e
6112024-10-22T00:31:06.016ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6122024-10-22T00:31:06.016ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6132024-10-22T00:31:06.017ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 5901186 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:62070 req_id = c28218c8-7ce2-40ff-a1bf-49bc271f85cf response_code = 204 uri = /omicron-zones
6142024-10-22T00:31:06.019ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6152024-10-22T00:31:06.049ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6162024-10-22T00:31:06.049ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48609
6172024-10-22T00:31:06.058ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3914
6182024-10-22T00:31:06.058ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 9270 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48609 req_id = 139a5b19-8eef-4cec-96b1-0329bedfd5a6 response_code = 500 uri = /timesync
6192024-10-22T00:31:06.060ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6202024-10-22T00:31:06.201ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6212024-10-22T00:31:06.201ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41708
6222024-10-22T00:31:07.084ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8308348012536343278 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
6232024-10-22T00:31:07.084ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2561833297137706320 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
6242024-10-22T00:31:07.589ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12602934267546915325 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
6252024-10-22T00:31:07.590ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4098285966382031446 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
6262024-10-22T00:31:09.492ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 605788162517344797 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
6272024-10-22T00:31:09.807ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5238568977474308989 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
6282024-10-22T00:31:09.911ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7815033753748808378 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
6292024-10-22T00:31:10.211ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3914
6302024-10-22T00:31:10.211ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4009667 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41708 req_id = e5b71841-9325-4848-8f40-917cedc40fcf response_code = 500 uri = /timesync
6312024-10-22T00:31:10.213ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6322024-10-22T00:31:10.379ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6332024-10-22T00:31:10.380ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50573
6342024-10-22T00:31:10.388ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3914
6352024-10-22T00:31:10.388ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8541 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50573 req_id = 8f28010f-d933-4c7c-b654-667620a6c54a response_code = 500 uri = /timesync
6362024-10-22T00:31:10.391ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6372024-10-22T00:31:10.623ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6382024-10-22T00:31:10.623ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52142
6392024-10-22T00:31:10.631ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_7db72912-69be-4df2-ab6e-2dd45407474e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3914
6402024-10-22T00:31:10.631ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52142 req_id = 6a36cf59-e840-4646-9cb0-ad938203e23e response_code = 500 uri = /timesync
6412024-10-22T00:31:10.634ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6422024-10-22T00:31:10.851ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6432024-10-22T00:31:10.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56324
6442024-10-22T00:31:14.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4008128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56324 req_id = 81809d98-af30-4e81-95c6-2051ede8a075 response_code = 200 uri = /timesync
6452024-10-22T00:31:14.866ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2180654074, ip_addr: 129.250.35.250, stratum: 3, ref_time: 1729557073.33709, correction: 0.05056015 } file = sled-agent/src/rack_setup/service.rs:660
6462024-10-22T00:31:14.866ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6472024-10-22T00:31:15.052ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6482024-10-22T00:31:15.053ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46731
6492024-10-22T00:31:15.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8357 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46731 req_id = b40ddb80-ca47-46dc-ab95-b1d37c868a93 response_code = 200 uri = /timesync
6502024-10-22T00:31:15.063ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2180654074, ip_addr: 129.250.35.250, stratum: 3, ref_time: 1729557073.33709, correction: 0.050026655 } file = sled-agent/src/rack_setup/service.rs:660
6512024-10-22T00:31:15.063ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6522024-10-22T00:31:15.366ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6532024-10-22T00:31:15.366ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38224
6542024-10-22T00:31:15.388ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 21868 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38224 req_id = 8fbc5cce-84e1-46bb-8e8b-26bed2cdf0b2 response_code = 200 uri = /timesync
6552024-10-22T00:31:15.390ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2180654074, ip_addr: 129.250.35.250, stratum: 3, ref_time: 1729557074.657946, correction: 0.049348019 } file = sled-agent/src/rack_setup/service.rs:660
6562024-10-22T00:31:15.390ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1298
6572024-10-22T00:31:15.419ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 7db72912-69be-4df2-ab6e-2dd45407474e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 7934b6c4-cfc3-4e03-b8b0-2ad4112eaa3b, kind: Service { id: 7db72912-69be-4df2-ab6e-2dd45407474e }, name: Name("ntp-7db72912-69be-4df2-ab6e-2dd45407474e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 146, 28])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 778b5da2-234f-4f22-9088-122f6000a2a4 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 51719187-c423-4136-bd14-0ac7604691ab (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } } } }, OmicronZoneConfig { id: 10ef867a-2629-4ba6-9747-a38a752b2d3f (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } } } }, OmicronZoneConfig { id: 614a56f5-ff81-44af-a8af-9fed369c7f01 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External } } } }, OmicronZoneConfig { id: af9278e2-efee-4d04-8e57-d084da405968 (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6582024-10-22T00:31:15.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63958
6592024-10-22T00:31:15.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6602024-10-22T00:31:15.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6612024-10-22T00:31:15.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6622024-10-22T00:31:15.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }
6632024-10-22T00:31:15.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6642024-10-22T00:31:15.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }
6652024-10-22T00:31:15.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 778b5da2-234f-4f22-9088-122f6000a2a4, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6662024-10-22T00:31:15.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 778b5da2-234f-4f22-9088-122f6000a2a4, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6672024-10-22T00:31:15.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 51719187-c423-4136-bd14-0ac7604691ab, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6682024-10-22T00:31:15.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 51719187-c423-4136-bd14-0ac7604691ab, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: Cockroach }, responder: ... }
6692024-10-22T00:31:15.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10ef867a-2629-4ba6-9747-a38a752b2d3f, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6702024-10-22T00:31:15.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10ef867a-2629-4ba6-9747-a38a752b2d3f, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: Cockroach }, responder: ... }
6712024-10-22T00:31:15.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 614a56f5-ff81-44af-a8af-9fed369c7f01, dataset_name: DatasetName { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6722024-10-22T00:31:15.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 614a56f5-ff81-44af-a8af-9fed369c7f01, dataset_name: DatasetName { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6732024-10-22T00:31:15.685ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: af9278e2-efee-4d04-8e57-d084da405968, dataset_name: DatasetName { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6742024-10-22T00:31:15.685ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: af9278e2-efee-4d04-8e57-d084da405968, dataset_name: DatasetName { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6752024-10-22T00:31:15.736ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6762024-10-22T00:31:15.736ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-zones.json file = common/src/ledger.rs:177
6772024-10-22T00:31:15.736ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-zones.json file = common/src/ledger.rs:177
6782024-10-22T00:31:15.736ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6792024-10-22T00:31:15.817ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6802024-10-22T00:31:15.817ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6812024-10-22T00:31:15.817ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6822024-10-22T00:31:15.817ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6832024-10-22T00:31:15.817ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6842024-10-22T00:31:15.906ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4 file = illumos-utils/src/zone.rs:316
6852024-10-22T00:31:15.910ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01 file = illumos-utils/src/zone.rs:316
6862024-10-22T00:31:15.915ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab file = illumos-utils/src/zone.rs:316
6872024-10-22T00:31:15.919ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968 file = illumos-utils/src/zone.rs:316
6882024-10-22T00:31:15.923ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f file = illumos-utils/src/zone.rs:316
6892024-10-22T00:31:15.927ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4 file = illumos-utils/src/zone.rs:354
6902024-10-22T00:31:15.932ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01 file = illumos-utils/src/zone.rs:354
6912024-10-22T00:31:15.937ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab file = illumos-utils/src/zone.rs:354
6922024-10-22T00:31:15.942ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968 file = illumos-utils/src/zone.rs:354
6932024-10-22T00:31:15.947ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f file = illumos-utils/src/zone.rs:354
6942024-10-22T00:31:18.910ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='51719187-c423-4136-bd14-0ac7604691ab'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6952024-10-22T00:31:18.910ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='10ef867a-2629-4ba6-9747-a38a752b2d3f'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6962024-10-22T00:31:18.910ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
6972024-10-22T00:31:18.915ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
6982024-10-22T00:31:18.983ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='778b5da2-234f-4f22-9088-122f6000a2a4'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6992024-10-22T00:31:18.983ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='614a56f5-ff81-44af-a8af-9fed369c7f01'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7002024-10-22T00:31:18.983ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7012024-10-22T00:31:18.988ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7022024-10-22T00:31:19.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 72.108752ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7032024-10-22T00:31:19.061ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='af9278e2-efee-4d04-8e57-d084da405968'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7042024-10-22T00:31:19.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 29.007126ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7052024-10-22T00:31:19.070ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7062024-10-22T00:31:19.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 144.372591ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7072024-10-22T00:31:19.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 25.119999ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7082024-10-22T00:31:19.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 70.536989ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7092024-10-22T00:31:19.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 52.149015ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7102024-10-22T00:31:19.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 64.158575ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7112024-10-22T00:31:20.029ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 270.180397ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
7122024-10-22T00:31:20.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 279.626313ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7132024-10-22T00:31:20.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 134.438885ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7142024-10-22T00:31:21.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 175.816584ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7152024-10-22T00:31:21.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 66.126477ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7162024-10-22T00:31:21.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 153.758753ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7172024-10-22T00:31:21.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 282.953711ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7182024-10-22T00:31:21.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 210.946664ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7192024-10-22T00:31:21.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 354.887322ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7202024-10-22T00:31:21.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 116.123819ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7212024-10-22T00:31:21.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 446.741718ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7222024-10-22T00:31:21.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 169.256103ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7232024-10-22T00:31:21.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 543.219075ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7242024-10-22T00:31:21.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 807.044092ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7252024-10-22T00:31:21.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 556.169339ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7262024-10-22T00:31:21.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 503.208551ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7272024-10-22T00:31:22.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 964.749441ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7282024-10-22T00:31:22.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 1.075762659s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7292024-10-22T00:31:22.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 1.25307053s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7302024-10-22T00:31:22.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 577.938108ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7312024-10-22T00:31:22.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 909.864179ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7322024-10-22T00:31:22.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 1.342770635s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7332024-10-22T00:31:23.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 783.971844ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7342024-10-22T00:31:23.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 1.369108842s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7352024-10-22T00:31:23.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 1.331674909s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7362024-10-22T00:31:23.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 1.21464973s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7372024-10-22T00:31:23.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 1.173442161s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7382024-10-22T00:31:24.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 1.301198901s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7392024-10-22T00:31:24.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 1.024639994s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7402024-10-22T00:31:24.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 1.248531713s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7412024-10-22T00:31:24.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 608.153569ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7422024-10-22T00:31:25.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 1.301074708s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7432024-10-22T00:31:25.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 1.052118019s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7442024-10-22T00:31:25.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 637.08053ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7452024-10-22T00:31:25.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 711.400242ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7462024-10-22T00:31:26.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 1.164518453s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7472024-10-22T00:31:26.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 543.342368ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7482024-10-22T00:31:26.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 587.833429ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7492024-10-22T00:31:26.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 733.691714ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7502024-10-22T00:31:26.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 692.703634ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7512024-10-22T00:31:26.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 540.460565ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7522024-10-22T00:31:26.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 1.17763813s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7532024-10-22T00:31:27.098ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48696
7542024-10-22T00:31:27.098ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48696 req_id = c165358c-6ece-4221-8f90-c2077e0ca7b5 response_code = 200 uri = /baseboard
7552024-10-22T00:31:27.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 1.39689615s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7562024-10-22T00:31:27.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 1.119921858s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7572024-10-22T00:31:27.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 1.363086163s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7582024-10-22T00:31:27.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 1.297386881s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7592024-10-22T00:31:28.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 541.25012ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7602024-10-22T00:31:28.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 818.434729ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7612024-10-22T00:31:28.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 944.976894ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7622024-10-22T00:31:28.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 1.348144782s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7632024-10-22T00:31:28.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 916.824224ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7642024-10-22T00:31:28.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 1.122659295s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7652024-10-22T00:31:29.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 1.13745391s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7662024-10-22T00:31:29.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 1.345646745s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7672024-10-22T00:31:29.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 1.173790586s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7682024-10-22T00:31:29.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 902.051421ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7692024-10-22T00:31:30.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 1.347117512s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7702024-10-22T00:31:30.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 1.343716501s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7712024-10-22T00:31:30.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 563.142633ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7722024-10-22T00:31:30.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 821.819565ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7732024-10-22T00:31:30.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 1.042538122s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7742024-10-22T00:31:31.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 578.647473ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7752024-10-22T00:31:31.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4") failed: Property not found. retry in 1.17119511s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7762024-10-22T00:31:31.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01") failed: Property not found. retry in 703.966694ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7772024-10-22T00:31:31.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab") failed: Property not found. retry in 779.326117ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7782024-10-22T00:31:31.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968") failed: Property not found. retry in 962.970093ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7792024-10-22T00:31:32.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f") failed: Property not found. retry in 1.001362688s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7802024-10-22T00:31:33.052ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_cockroachdb_778b5da2-234f-4f22-9088-122f6000a2a4
7812024-10-22T00:31:33.053ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_cockroachdb_614a56f5-ff81-44af-a8af-9fed369c7f01
7822024-10-22T00:31:33.053ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_cockroachdb_51719187-c423-4136-bd14-0ac7604691ab
7832024-10-22T00:31:33.053ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_cockroachdb_af9278e2-efee-4d04-8e57-d084da405968
7842024-10-22T00:31:33.053ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_cockroachdb_10ef867a-2629-4ba6-9747-a38a752b2d3f
7852024-10-22T00:31:33.053ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7862024-10-22T00:31:33.053ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7872024-10-22T00:31:33.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 17586078 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:63958 req_id = 98ac7a5b-d7f6-464a-9e7f-02d3f3eb358e response_code = 204 uri = /omicron-zones
7882024-10-22T00:31:33.086ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55694
7892024-10-22T00:31:33.111ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:3757
7902024-10-22T00:31:36.565ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 301.607276ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
7912024-10-22T00:31:51.530ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3775
7922024-10-22T00:31:51.900ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.459679529s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
7932024-10-22T00:31:55.363ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3798
7942024-10-22T00:31:55.363ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22276726 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:55694 req_id = e01095df-78fe-435f-b8aa-564e933beb6a response_code = 204 uri = /cockroachdb
7952024-10-22T00:31:55.395ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 7db72912-69be-4df2-ab6e-2dd45407474e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 7934b6c4-cfc3-4e03-b8b0-2ad4112eaa3b, kind: Service { id: 7db72912-69be-4df2-ab6e-2dd45407474e }, name: Name("ntp-7db72912-69be-4df2-ab6e-2dd45407474e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 146, 28])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 778b5da2-234f-4f22-9088-122f6000a2a4 (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 51719187-c423-4136-bd14-0ac7604691ab (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } } } }, OmicronZoneConfig { id: 10ef867a-2629-4ba6-9747-a38a752b2d3f (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } } } }, OmicronZoneConfig { id: 614a56f5-ff81-44af-a8af-9fed369c7f01 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External } } } }, OmicronZoneConfig { id: af9278e2-efee-4d04-8e57-d084da405968 (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External } } } }, OmicronZoneConfig { id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 0a9617e4-b230-4cf3-bf60-222f1daf4df2, kind: Service { id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320 }, name: Name("external-dns-e8c37ad4-fd39-4460-b5dc-d6c95ee10320"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 220])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 68a0f950-6467-48bb-b198-77a5253fae12, kind: Service { id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb }, name: Name("external-dns-1ed57c3f-d967-4545-97bd-c26e5c30bbbb"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 169])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: ac50d2b2-d7d9-4d55-9c28-567f3362fbbb (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 60b8c23e-4f5b-4259-9ee0-0845046b29a4, kind: Service { id: ac50d2b2-d7d9-4d55-9c28-567f3362fbbb }, name: Name("nexus-ac50d2b2-d7d9-4d55-9c28-567f3362fbbb"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 115])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b360f3f1-c2e5-43e4-b254-fe92a717e371 (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 73b1ae2e-5e7d-4c5e-8af0-31d8145c570a, kind: Service { id: b360f3f1-c2e5-43e4-b254-fe92a717e371 }, name: Name("nexus-b360f3f1-c2e5-43e4-b254-fe92a717e371"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 21])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1ddbaee3-3dbf-4414-b1d3-075d046a666f (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ec446109-19ba-4097-9ed2-643d63373744, kind: Service { id: 1ddbaee3-3dbf-4414-b1d3-075d046a666f }, name: Name("nexus-1ddbaee3-3dbf-4414-b1d3-075d046a666f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 183])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f7ab276a-527a-42f6-b169-e0079c024430 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: c4e4554d-3767-4139-9cd6-7b8f7bc20696 (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 26cb4910-66e0-447b-a9fe-b56f1eccd33e (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ef33f36a-2317-4f47-821d-4cfbccd3056c (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: b6fabd16-0c44-4d4e-8891-220660ced37f (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: bc227b7a-504b-4d14-a0b2-298387ecf3d7 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14bbe5e4-bb66-4b10-abdd-57d4f73f950e (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External } } } }, OmicronZoneConfig { id: 480888f2-5a14-4872-84ee-4203b8e8ad3b (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External } } } }, OmicronZoneConfig { id: cc9c640a-d082-4992-ae26-2265af436049 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External } } } }, OmicronZoneConfig { id: e25dacf9-31af-4674-910b-dafdcdcd38d8 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External } } } }, OmicronZoneConfig { id: 960438c0-57fe-4406-a0be-4f4ee6b84a88 (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External } } } }, OmicronZoneConfig { id: df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1b0d34b4-83ba-4bc1-974b-4b8176749e06 (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7962024-10-22T00:31:55.395ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42112
7972024-10-22T00:31:55.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7982024-10-22T00:31:55.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 927912b9-9e81-49b4-8c6d-401ffc3d66ad, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7992024-10-22T00:31:55.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8002024-10-22T00:31:55.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 42eaca0b-e1f8-45e1-96a4-db2b312ecbf7, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: InternalDns }, responder: ... }
8012024-10-22T00:31:55.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8022024-10-22T00:31:55.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3a7586a1-eab7-4d3d-af3d-cc4036fbeb5d, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: InternalDns }, responder: ... }
8032024-10-22T00:31:55.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 778b5da2-234f-4f22-9088-122f6000a2a4, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8042024-10-22T00:31:55.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 778b5da2-234f-4f22-9088-122f6000a2a4, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8052024-10-22T00:31:55.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 51719187-c423-4136-bd14-0ac7604691ab, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8062024-10-22T00:31:55.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 51719187-c423-4136-bd14-0ac7604691ab, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: Cockroach }, responder: ... }
8072024-10-22T00:31:55.499ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10ef867a-2629-4ba6-9747-a38a752b2d3f, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8082024-10-22T00:31:55.499ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10ef867a-2629-4ba6-9747-a38a752b2d3f, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: Cockroach }, responder: ... }
8092024-10-22T00:31:55.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 614a56f5-ff81-44af-a8af-9fed369c7f01, dataset_name: DatasetName { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8102024-10-22T00:31:55.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 614a56f5-ff81-44af-a8af-9fed369c7f01, dataset_name: DatasetName { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8112024-10-22T00:31:55.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: af9278e2-efee-4d04-8e57-d084da405968, dataset_name: DatasetName { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8122024-10-22T00:31:55.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: af9278e2-efee-4d04-8e57-d084da405968, dataset_name: DatasetName { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8132024-10-22T00:31:55.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8142024-10-22T00:31:55.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8152024-10-22T00:31:55.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8162024-10-22T00:31:55.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8172024-10-22T00:31:55.715ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c4e4554d-3767-4139-9cd6-7b8f7bc20696, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
8182024-10-22T00:31:55.715ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c4e4554d-3767-4139-9cd6-7b8f7bc20696, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8192024-10-22T00:31:55.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bc227b7a-504b-4d14-a0b2-298387ecf3d7, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8202024-10-22T00:31:55.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: bc227b7a-504b-4d14-a0b2-298387ecf3d7, dataset_name: DatasetName { pool_name: ZpoolName { id: 05815180-fa73-47c8-8793-9dad48a6a2a3 (zpool), kind: External }, kind: Crucible }, responder: ... }
8212024-10-22T00:31:55.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14bbe5e4-bb66-4b10-abdd-57d4f73f950e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8222024-10-22T00:31:55.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 14bbe5e4-bb66-4b10-abdd-57d4f73f950e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4912870c-99c9-4864-9312-2d393933ce6a (zpool), kind: External }, kind: Crucible }, responder: ... }
8232024-10-22T00:31:55.959ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 480888f2-5a14-4872-84ee-4203b8e8ad3b, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8242024-10-22T00:31:55.959ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 480888f2-5a14-4872-84ee-4203b8e8ad3b, dataset_name: DatasetName { pool_name: ZpoolName { id: f643b80c-1fc2-4be9-93a9-fa7fd37e716e (zpool), kind: External }, kind: Crucible }, responder: ... }
8252024-10-22T00:31:56.039ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc9c640a-d082-4992-ae26-2265af436049, dataset_name: DatasetName { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8262024-10-22T00:31:56.039ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: cc9c640a-d082-4992-ae26-2265af436049, dataset_name: DatasetName { pool_name: ZpoolName { id: f3053f19-a930-43cc-8978-6dcb10f47d61 (zpool), kind: External }, kind: Crucible }, responder: ... }
8272024-10-22T00:31:56.123ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62, dataset_name: DatasetName { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8282024-10-22T00:31:56.123ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62, dataset_name: DatasetName { pool_name: ZpoolName { id: d912c6f5-c454-43da-ae16-2d232f5f0eb5 (zpool), kind: External }, kind: Crucible }, responder: ... }
8292024-10-22T00:31:56.200ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e25dacf9-31af-4674-910b-dafdcdcd38d8, dataset_name: DatasetName { pool_name: ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8302024-10-22T00:31:56.200ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e25dacf9-31af-4674-910b-dafdcdcd38d8, dataset_name: DatasetName { pool_name: ZpoolName { id: f72e2a71-8f15-4409-923c-af354b7632aa (zpool), kind: External }, kind: Crucible }, responder: ... }
8312024-10-22T00:31:56.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 960438c0-57fe-4406-a0be-4f4ee6b84a88, dataset_name: DatasetName { pool_name: ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8322024-10-22T00:31:56.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 960438c0-57fe-4406-a0be-4f4ee6b84a88, dataset_name: DatasetName { pool_name: ZpoolName { id: 4124bd0f-6d92-452d-b8ec-890efde7c175 (zpool), kind: External }, kind: Crucible }, responder: ... }
8332024-10-22T00:31:56.337ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8342024-10-22T00:31:56.337ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bea9bba-836b-44a8-977b-0b50b3178081 (zpool), kind: External }, kind: Crucible }, responder: ... }
8352024-10-22T00:31:56.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1b0d34b4-83ba-4bc1-974b-4b8176749e06, dataset_name: DatasetName { pool_name: ZpoolName { id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8362024-10-22T00:31:56.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 1b0d34b4-83ba-4bc1-974b-4b8176749e06, dataset_name: DatasetName { pool_name: ZpoolName { id: c2c09b26-0ffd-409b-9a74-0a32dc4478f3 (zpool), kind: External }, kind: Crucible }, responder: ... }
8372024-10-22T00:31:56.464ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8382024-10-22T00:31:56.464ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-zones.json file = common/src/ledger.rs:177
8392024-10-22T00:31:56.464ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-zones.json file = common/src/ledger.rs:177
8402024-10-22T00:31:56.464ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8412024-10-22T00:31:56.563ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
8422024-10-22T00:31:56.587ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
8432024-10-22T00:31:56.587ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8442024-10-22T00:31:56.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8452024-10-22T00:31:56.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8462024-10-22T00:31:56.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8472024-10-22T00:31:56.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8482024-10-22T00:31:56.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8492024-10-22T00:31:56.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8502024-10-22T00:31:56.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8512024-10-22T00:31:56.808ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
8522024-10-22T00:31:56.808ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8532024-10-22T00:31:56.808ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8542024-10-22T00:31:56.820ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
8552024-10-22T00:31:56.878ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
8562024-10-22T00:31:56.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8572024-10-22T00:31:56.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8582024-10-22T00:31:56.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8592024-10-22T00:31:56.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8602024-10-22T00:31:56.879ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6 file = illumos-utils/src/zone.rs:316
8612024-10-22T00:31:56.885ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430 file = illumos-utils/src/zone.rs:316
8622024-10-22T00:31:56.890ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e file = illumos-utils/src/zone.rs:316
8632024-10-22T00:31:56.897ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c file = illumos-utils/src/zone.rs:316
8642024-10-22T00:31:56.903ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f file = illumos-utils/src/zone.rs:316
8652024-10-22T00:31:56.909ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7 file = illumos-utils/src/zone.rs:316
8662024-10-22T00:31:56.915ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049 file = illumos-utils/src/zone.rs:316
8672024-10-22T00:31:56.921ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e file = illumos-utils/src/zone.rs:316
8682024-10-22T00:31:57.045ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6 file = illumos-utils/src/zone.rs:354
8692024-10-22T00:31:57.049ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430 file = illumos-utils/src/zone.rs:354
8702024-10-22T00:31:57.054ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e file = illumos-utils/src/zone.rs:354
8712024-10-22T00:31:57.059ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c file = illumos-utils/src/zone.rs:354
8722024-10-22T00:31:57.063ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f file = illumos-utils/src/zone.rs:354
8732024-10-22T00:31:57.068ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7 file = illumos-utils/src/zone.rs:354
8742024-10-22T00:31:57.073ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049 file = illumos-utils/src/zone.rs:354
8752024-10-22T00:31:57.078ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e file = illumos-utils/src/zone.rs:354
8762024-10-22T00:31:57.082ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62 file = illumos-utils/src/zone.rs:316
8772024-10-22T00:31:57.087ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8 file = illumos-utils/src/zone.rs:316
8782024-10-22T00:31:57.093ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88 file = illumos-utils/src/zone.rs:316
8792024-10-22T00:31:57.098ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06 file = illumos-utils/src/zone.rs:316
8802024-10-22T00:31:57.104ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696 file = illumos-utils/src/zone.rs:316
8812024-10-22T00:31:57.111ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b file = illumos-utils/src/zone.rs:316
8822024-10-22T00:31:57.118ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62 file = illumos-utils/src/zone.rs:354
8832024-10-22T00:31:57.123ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8 file = illumos-utils/src/zone.rs:354
8842024-10-22T00:31:57.129ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88 file = illumos-utils/src/zone.rs:354
8852024-10-22T00:31:57.135ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06 file = illumos-utils/src/zone.rs:354
8862024-10-22T00:31:57.142ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696 file = illumos-utils/src/zone.rs:354
8872024-10-22T00:31:57.148ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b file = illumos-utils/src/zone.rs:354
8882024-10-22T00:31:57.166ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:57356
8892024-10-22T00:31:57.166ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:57356 req_id = b35bb778-ad2d-4656-b2fb-fe3ba8dd7dd3 response_code = 200 uri = /baseboard
8902024-10-22T00:31:57.207ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
8912024-10-22T00:31:57.291ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
8922024-10-22T00:31:57.363ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
8932024-10-22T00:31:57.427ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
8942024-10-22T00:31:57.520ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
8952024-10-22T00:31:57.523ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
8962024-10-22T00:31:57.523ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
8972024-10-22T00:31:57.581ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 150, 183]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
8982024-10-22T00:31:57.581ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1270 zone_type = nexus
8992024-10-22T00:31:57.582ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
9002024-10-22T00:31:57.583ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
9012024-10-22T00:31:57.724ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 229, 21]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9022024-10-22T00:31:57.724ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1270 zone_type = nexus
9032024-10-22T00:31:57.727ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
9042024-10-22T00:31:57.727ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
9052024-10-22T00:31:57.797ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 182, 169]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
9062024-10-22T00:31:57.797ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1270 zone_type = external_dns
9072024-10-22T00:31:57.812ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
9082024-10-22T00:31:57.812ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
9092024-10-22T00:31:57.896ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 186, 220]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
9102024-10-22T00:31:57.896ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1270 zone_type = external_dns
9112024-10-22T00:31:57.899ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
9122024-10-22T00:31:57.899ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
9132024-10-22T00:31:57.960ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 217, 115]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9142024-10-22T00:31:57.961ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1270 zone_type = nexus
9152024-10-22T00:31:57.961ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
9162024-10-22T00:31:57.961ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
9172024-10-22T00:31:57.961ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
9182024-10-22T00:31:57.961ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
9192024-10-22T00:31:57.962ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
9202024-10-22T00:31:57.964ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9212024-10-22T00:31:58.019ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
9222024-10-22T00:31:58.019ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
9232024-10-22T00:31:58.020ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
9242024-10-22T00:31:58.020ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
9252024-10-22T00:31:58.020ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
9262024-10-22T00:31:58.020ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
9272024-10-22T00:31:58.020ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
9282024-10-22T00:31:58.023ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9292024-10-22T00:31:58.023ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
9302024-10-22T00:31:58.028ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
9312024-10-22T00:31:58.031ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
9322024-10-22T00:31:58.031ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9332024-10-22T00:31:58.031ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9342024-10-22T00:31:58.085ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9352024-10-22T00:31:58.085ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f file = illumos-utils/src/zone.rs:316
9362024-10-22T00:31:58.196ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371 file = illumos-utils/src/zone.rs:316
9372024-10-22T00:31:58.203ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f file = illumos-utils/src/zone.rs:354
9382024-10-22T00:31:58.211ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb file = illumos-utils/src/zone.rs:316
9392024-10-22T00:31:58.220ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb file = illumos-utils/src/zone.rs:316
9402024-10-22T00:31:58.227ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320 file = illumos-utils/src/zone.rs:316
9412024-10-22T00:31:58.236ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371 file = illumos-utils/src/zone.rs:354
9422024-10-22T00:31:58.248ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb file = illumos-utils/src/zone.rs:354
9432024-10-22T00:31:58.267ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb file = illumos-utils/src/zone.rs:354
9442024-10-22T00:31:58.284ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320 file = illumos-utils/src/zone.rs:354
9452024-10-22T00:32:01.086ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9462024-10-22T00:32:01.086ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
9472024-10-22T00:32:01.110ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9482024-10-22T00:32:01.110ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
9492024-10-22T00:32:01.143ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9502024-10-22T00:32:01.143ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
9512024-10-22T00:32:01.162ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f3053f19-a930-43cc-8978-6dcb10f47d61/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3c7d8626-ab13-47b7-bd59-a0f4189b5745'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9522024-10-22T00:32:01.162ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c2c09b26-0ffd-409b-9a74-0a32dc4478f3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='8886178f-6190-4092-a955-8d9c18d78d5c'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9532024-10-22T00:32:01.162ZINFOSledAgent (ServiceManager): Booting oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
9542024-10-22T00:32:01.170ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_4124bd0f-6d92-452d-b8ec-890efde7c175/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='126e4733-c3ca-4154-92a9-7d6e7b2ae38a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9552024-10-22T00:32:01.170ZINFOSledAgent (ServiceManager): Booting oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
9562024-10-22T00:32:01.177ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f643b80c-1fc2-4be9-93a9-fa7fd37e716e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='8eac9e88-78d4-4091-af91-ef38532be887'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9572024-10-22T00:32:01.177ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_4912870c-99c9-4864-9312-2d393933ce6a/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='ab87ce03-cb18-471e-910a-f6cdf524d048'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9582024-10-22T00:32:01.177ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_d912c6f5-c454-43da-ae16-2d232f5f0eb5/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3cb10bb1-1544-450d-99b6-d368aa8bbdc1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9592024-10-22T00:32:01.178ZINFOSledAgent (ServiceManager): Booting oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
9602024-10-22T00:32:01.184ZINFOSledAgent (ServiceManager): Booting oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
9612024-10-22T00:32:01.190ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_6bea9bba-836b-44a8-977b-0b50b3178081/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='8fed7f07-9056-43ca-ad47-ab253e16cb98'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9622024-10-22T00:32:01.190ZINFOSledAgent (ServiceManager): Booting oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
9632024-10-22T00:32:01.196ZINFOSledAgent (ServiceManager): Booting oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
9642024-10-22T00:32:01.202ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f72e2a71-8f15-4409-923c-af354b7632aa/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a1d8af9c-a3f9-47cc-b098-926cadbd9081'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9652024-10-22T00:32:01.202ZINFOSledAgent (ServiceManager): Booting oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
9662024-10-22T00:32:01.209ZINFOSledAgent (ServiceManager): Booting oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
9672024-10-22T00:32:01.238ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_05815180-fa73-47c8-8793-9dad48a6a2a3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d936fc91-4083-47fa-8ff6-0374010aa369'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9682024-10-22T00:32:01.238ZINFOSledAgent (ServiceManager): Booting oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
9692024-10-22T00:32:01.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 65.620984ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
9702024-10-22T00:32:01.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 44.821944ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
9712024-10-22T00:32:01.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 29.792972ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
9722024-10-22T00:32:01.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 34.214161ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
9732024-10-22T00:32:01.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 29.917579ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
9742024-10-22T00:32:01.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 54.875131ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
9752024-10-22T00:32:01.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 37.210259ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
9762024-10-22T00:32:01.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 53.605878ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
9772024-10-22T00:32:01.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 31.916766ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
9782024-10-22T00:32:01.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 125.488763ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
9792024-10-22T00:32:01.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 136.74885ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
9802024-10-22T00:32:01.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 68.282378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
9812024-10-22T00:32:04.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 105.461293ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
9822024-10-22T00:32:04.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 64.646436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
9832024-10-22T00:32:04.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 140.748971ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
9842024-10-22T00:32:04.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 35.409567ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
9852024-10-22T00:32:04.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 89.30227ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
9862024-10-22T00:32:04.954ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='f7ab276a-527a-42f6-b169-e0079c024430'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9872024-10-22T00:32:05.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 81.68351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
9882024-10-22T00:32:05.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 147.087677ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
9892024-10-22T00:32:05.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 164.004936ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
9902024-10-22T00:32:05.541ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 129.735741ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
9912024-10-22T00:32:05.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 71.037585ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
9922024-10-22T00:32:05.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 193.140266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
9932024-10-22T00:32:06.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 85.553788ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
9942024-10-22T00:32:06.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 137.199092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
9952024-10-22T00:32:06.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 109.77745ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
9962024-10-22T00:32:06.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 100.329932ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
9972024-10-22T00:32:06.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 126.313254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
9982024-10-22T00:32:06.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 214.12625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
9992024-10-22T00:32:06.467ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10002024-10-22T00:32:06.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 290.895875ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
10012024-10-22T00:32:06.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 485.850461ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
10022024-10-22T00:32:06.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 583.4447ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
10032024-10-22T00:32:06.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 244.865995ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
10042024-10-22T00:32:06.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 166.364956ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
10052024-10-22T00:32:06.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 356.19255ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
10062024-10-22T00:32:06.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 216.685923ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
10072024-10-22T00:32:06.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 188.023817ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
10082024-10-22T00:32:06.808ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 171.589644ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
10092024-10-22T00:32:06.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 328.405545ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
10102024-10-22T00:32:06.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 147.358749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
10112024-10-22T00:32:06.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 549.689415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
10122024-10-22T00:32:06.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 299.118883ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
10132024-10-22T00:32:06.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 358.576765ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
10142024-10-22T00:32:06.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 493.401839ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
10152024-10-22T00:32:06.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 432.034877ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
10162024-10-22T00:32:07.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 285.071371ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
10172024-10-22T00:32:07.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 275.587329ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
10182024-10-22T00:32:07.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 49.646744ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10192024-10-22T00:32:07.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 381.662519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
10202024-10-22T00:32:07.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 140.261146ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10212024-10-22T00:32:07.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 522.118668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
10222024-10-22T00:32:07.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 417.039417ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
10232024-10-22T00:32:07.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 909.154889ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
10242024-10-22T00:32:07.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 499.254718ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
10252024-10-22T00:32:07.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 133.633286ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10262024-10-22T00:32:07.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.184815153s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
10272024-10-22T00:32:07.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 413.927436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
10282024-10-22T00:32:07.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.023725093s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
10292024-10-22T00:32:07.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 412.974059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
10302024-10-22T00:32:07.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 291.722667ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10312024-10-22T00:32:07.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 614.067023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
10322024-10-22T00:32:07.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 840.537215ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
10332024-10-22T00:32:07.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 709.770757ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
10342024-10-22T00:32:07.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 969.349657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
10352024-10-22T00:32:07.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.10017267s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
10362024-10-22T00:32:07.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 523.146625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
10372024-10-22T00:32:07.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 849.152949ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10382024-10-22T00:32:07.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.444985813s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
10392024-10-22T00:32:07.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.353579753s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
10402024-10-22T00:32:07.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 589.107904ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
10412024-10-22T00:32:08.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.194015963s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
10422024-10-22T00:32:08.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 927.506362ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
10432024-10-22T00:32:08.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.108902344s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
10442024-10-22T00:32:08.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 549.809958ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
10452024-10-22T00:32:08.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.484103836s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
10462024-10-22T00:32:08.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.120919927s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
10472024-10-22T00:32:08.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.033409775s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
10482024-10-22T00:32:08.429ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.307511068s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
10492024-10-22T00:32:08.443ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10502024-10-22T00:32:08.443ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
10512024-10-22T00:32:08.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 726.428078ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
10522024-10-22T00:32:08.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.089129165s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
10532024-10-22T00:32:08.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 981.993965ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10542024-10-22T00:32:08.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 841.799033ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
10552024-10-22T00:32:08.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 511.124945ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
10562024-10-22T00:32:08.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 67.444417ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
10572024-10-22T00:32:08.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 66.889077ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
10582024-10-22T00:32:08.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 284.077147ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
10592024-10-22T00:32:09.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 933.914764ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
10602024-10-22T00:32:09.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 630.644518ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
10612024-10-22T00:32:09.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 868.969566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
10622024-10-22T00:32:09.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 896.362322ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
10632024-10-22T00:32:09.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 378.806427ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
10642024-10-22T00:32:09.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.222192935s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
10652024-10-22T00:32:09.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.329336469s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
10662024-10-22T00:32:09.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.132153639s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
10672024-10-22T00:32:09.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 984.879345ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
10682024-10-22T00:32:09.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 546.907104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
10692024-10-22T00:32:09.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.402688881s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
10702024-10-22T00:32:09.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.44388306s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
10712024-10-22T00:32:09.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 750.427394ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10722024-10-22T00:32:09.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 696.508533ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
10732024-10-22T00:32:09.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 645.027606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
10742024-10-22T00:32:09.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.043356423s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
10752024-10-22T00:32:09.920ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10762024-10-22T00:32:09.920ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10772024-10-22T00:32:09.920ZINFOSledAgent (ServiceManager): Booting oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
10782024-10-22T00:32:09.928ZINFOSledAgent (ServiceManager): Booting oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
10792024-10-22T00:32:09.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.23905045s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
10802024-10-22T00:32:10.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 766.151239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
10812024-10-22T00:32:10.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 884.613212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
10822024-10-22T00:32:10.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.274214074s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
10832024-10-22T00:32:10.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.486093749s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
10842024-10-22T00:32:10.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.010150255s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
10852024-10-22T00:32:10.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.230170067s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
10862024-10-22T00:32:10.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.231802084s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
10872024-10-22T00:32:10.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.309742692s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
10882024-10-22T00:32:10.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.258821347s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
10892024-10-22T00:32:11.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 25.03029ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
10902024-10-22T00:32:11.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 826.699295ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
10912024-10-22T00:32:11.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.157917698s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
10922024-10-22T00:32:11.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 74.908456ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
10932024-10-22T00:32:11.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 724.610186ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
10942024-10-22T00:32:11.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.455952871s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
10952024-10-22T00:32:11.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 934.046568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
10962024-10-22T00:32:11.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.32146022s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
10972024-10-22T00:32:11.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 95.160614ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
10982024-10-22T00:32:11.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 941.172999ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
10992024-10-22T00:32:11.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 63.576521ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11002024-10-22T00:32:11.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 988.955932ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
11012024-10-22T00:32:11.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.203566398s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
11022024-10-22T00:32:11.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 272.904979ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11032024-10-22T00:32:11.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 199.968312ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11042024-10-22T00:32:11.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 962.904919ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
11052024-10-22T00:32:11.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.344032092s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
11062024-10-22T00:32:12.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 236.634823ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11072024-10-22T00:32:12.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.344858842s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
11082024-10-22T00:32:12.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 450.992112ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11092024-10-22T00:32:12.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 993.149217ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
11102024-10-22T00:32:12.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 556.416556ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
11112024-10-22T00:32:12.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.292102781s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
11122024-10-22T00:32:12.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.293450862s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
11132024-10-22T00:32:12.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.443264662s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
11142024-10-22T00:32:12.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 734.581263ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
11152024-10-22T00:32:12.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.00951424s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11162024-10-22T00:32:12.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 836.75427ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11172024-10-22T00:32:12.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.231303869s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
11182024-10-22T00:32:12.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 675.352151ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
11192024-10-22T00:32:12.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 576.062552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
11202024-10-22T00:32:12.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.180073966s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
11212024-10-22T00:32:12.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.014318263s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
11222024-10-22T00:32:12.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.078545734s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
11232024-10-22T00:32:13.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.374394845s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
11242024-10-22T00:32:13.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.244121988s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
11252024-10-22T00:32:13.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.354478833s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
11262024-10-22T00:32:13.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 994.811221ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
11272024-10-22T00:32:13.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 705.92469ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
11282024-10-22T00:32:13.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.092482244s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
11292024-10-22T00:32:13.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.120574695s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11302024-10-22T00:32:13.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.396573317s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11312024-10-22T00:32:13.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.334564034s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
11322024-10-22T00:32:13.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.306791063s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
11332024-10-22T00:32:13.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 574.073084ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
11342024-10-22T00:32:13.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.194462371s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
11352024-10-22T00:32:13.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.190904536s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
11362024-10-22T00:32:13.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.482002161s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
11372024-10-22T00:32:13.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.019157262s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
11382024-10-22T00:32:13.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.114729775s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
11392024-10-22T00:32:14.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 672.216749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
11402024-10-22T00:32:14.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 904.675972ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
11412024-10-22T00:32:14.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.031651998s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
11422024-10-22T00:32:14.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.426565902s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
11432024-10-22T00:32:14.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 504.435944ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
11442024-10-22T00:32:14.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 613.331307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
11452024-10-22T00:32:14.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.032735775s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11462024-10-22T00:32:14.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.432290396s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
11472024-10-22T00:32:14.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 889.563687ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
11482024-10-22T00:32:14.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 519.27525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
11492024-10-22T00:32:14.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 729.77514ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
11502024-10-22T00:32:14.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.038963855s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11512024-10-22T00:32:14.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.026100422s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
11522024-10-22T00:32:14.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.4601723s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
11532024-10-22T00:32:14.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.466017967s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
11542024-10-22T00:32:15.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.430887894s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
11552024-10-22T00:32:15.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 755.431393ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
11562024-10-22T00:32:15.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.052043669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
11572024-10-22T00:32:15.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 877.994881ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
11582024-10-22T00:32:15.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.302412945s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
11592024-10-22T00:32:15.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.288665074s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
11602024-10-22T00:32:15.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.422044053s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
11612024-10-22T00:32:15.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 609.767468ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11622024-10-22T00:32:15.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 502.09715ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
11632024-10-22T00:32:15.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.472251302s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
11642024-10-22T00:32:15.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.448021072s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
11652024-10-22T00:32:15.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 940.072018ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
11662024-10-22T00:32:15.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 975.794383ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11672024-10-22T00:32:15.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.487652447s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
11682024-10-22T00:32:15.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.169051837s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
11692024-10-22T00:32:15.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 861.871242ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
11702024-10-22T00:32:16.064ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.209246451s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
11712024-10-22T00:32:16.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.366280193s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
11722024-10-22T00:32:16.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 908.213622ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11732024-10-22T00:32:16.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.189913403s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
11742024-10-22T00:32:16.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 749.423743ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
11752024-10-22T00:32:16.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.273560344s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
11762024-10-22T00:32:16.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.328472898s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
11772024-10-22T00:32:16.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 749.716036ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
11782024-10-22T00:32:16.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 669.990285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
11792024-10-22T00:32:16.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.121131449s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
11802024-10-22T00:32:16.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 893.724893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
11812024-10-22T00:32:16.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 552.00563ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11822024-10-22T00:32:17.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 816.70897ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
11832024-10-22T00:32:17.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.331583507s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
11842024-10-22T00:32:17.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.200125191s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
11852024-10-22T00:32:17.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 652.791584ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
11862024-10-22T00:32:17.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.464390225s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
11872024-10-22T00:32:17.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.105611582s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
11882024-10-22T00:32:17.398ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.327266266s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
11892024-10-22T00:32:17.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 913.687033ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
11902024-10-22T00:32:17.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 680.887944ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
11912024-10-22T00:32:17.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 853.445634ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
11922024-10-22T00:32:17.511ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 524.930725ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
11932024-10-22T00:32:17.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 844.517882ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
11942024-10-22T00:32:17.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.002247436s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
11952024-10-22T00:32:17.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.393849347s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
11962024-10-22T00:32:17.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.224393065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
11972024-10-22T00:32:17.891ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11982024-10-22T00:32:17.891ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
11992024-10-22T00:32:17.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.370662937s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
12002024-10-22T00:32:17.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 878.3034ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
12012024-10-22T00:32:17.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 705.724574ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
12022024-10-22T00:32:18.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.399955207s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
12032024-10-22T00:32:18.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 53.752858ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12042024-10-22T00:32:18.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 140.942955ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12052024-10-22T00:32:18.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 976.683459ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
12062024-10-22T00:32:18.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 256.609727ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12072024-10-22T00:32:18.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 567.766588ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
12082024-10-22T00:32:18.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.031883019s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
12092024-10-22T00:32:18.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 544.838485ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
12102024-10-22T00:32:18.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 658.993284ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
12112024-10-22T00:32:18.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.341528649s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
12122024-10-22T00:32:18.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 696.406ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
12132024-10-22T00:32:18.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 518.621759ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12142024-10-22T00:32:18.562ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12152024-10-22T00:32:18.562ZINFOSledAgent (ServiceManager): Booting oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12162024-10-22T00:32:18.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.064560801s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
12172024-10-22T00:32:18.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 872.26917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
12182024-10-22T00:32:18.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 39.98626ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12192024-10-22T00:32:18.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.089754841s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
12202024-10-22T00:32:18.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 85.117672ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12212024-10-22T00:32:18.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 813.917234ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
12222024-10-22T00:32:18.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 778.339536ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
12232024-10-22T00:32:18.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 234.45754ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12242024-10-22T00:32:18.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 911.205051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
12252024-10-22T00:32:18.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.275499082s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
12262024-10-22T00:32:19.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 567.576204ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
12272024-10-22T00:32:19.077ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.054305269s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12282024-10-22T00:32:19.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.455561016s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
12292024-10-22T00:32:19.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 258.541236ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12302024-10-22T00:32:19.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.079994836s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
12312024-10-22T00:32:19.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 646.081845ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
12322024-10-22T00:32:19.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 697.214047ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
12332024-10-22T00:32:19.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 587.137444ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
12342024-10-22T00:32:19.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.046262289s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12352024-10-22T00:32:19.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.434068229s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
12362024-10-22T00:32:19.457ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.033079082s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
12372024-10-22T00:32:19.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 524.295166ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
12382024-10-22T00:32:19.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 877.812264ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
12392024-10-22T00:32:19.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 773.931798ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
12402024-10-22T00:32:19.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.136539053s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
12412024-10-22T00:32:19.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 947.845712ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
12422024-10-22T00:32:19.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.394836122s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
12432024-10-22T00:32:19.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 819.85211ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
12442024-10-22T00:32:19.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.206203006s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
12452024-10-22T00:32:19.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.223047637s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
12462024-10-22T00:32:19.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.462776366s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
12472024-10-22T00:32:19.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.309686241s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
12482024-10-22T00:32:20.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.227382837s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
12492024-10-22T00:32:20.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 687.597393ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12502024-10-22T00:32:20.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.215900067s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
12512024-10-22T00:32:20.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 620.61409ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
12522024-10-22T00:32:20.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 523.731275ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
12532024-10-22T00:32:20.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 710.261509ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12542024-10-22T00:32:20.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.192076679s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
12552024-10-22T00:32:20.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.157661632s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
12562024-10-22T00:32:20.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 510.487098ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
12572024-10-22T00:32:20.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 541.474633ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
12582024-10-22T00:32:20.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.139740971s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
12592024-10-22T00:32:20.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 755.621578ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
12602024-10-22T00:32:20.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 823.227965ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12612024-10-22T00:32:20.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 902.727718ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
12622024-10-22T00:32:20.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 807.406224ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
12632024-10-22T00:32:20.872ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12642024-10-22T00:32:20.873ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
12652024-10-22T00:32:20.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.240120475s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
12662024-10-22T00:32:21.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 41.798292ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
12672024-10-22T00:32:21.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 600.934614ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
12682024-10-22T00:32:21.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.249873069s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
12692024-10-22T00:32:21.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.081008424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
12702024-10-22T00:32:21.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 119.27231ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
12712024-10-22T00:32:21.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.335036244s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
12722024-10-22T00:32:21.184ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 974.590081ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
12732024-10-22T00:32:21.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.323812334s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
12742024-10-22T00:32:21.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 814.568327ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
12752024-10-22T00:32:21.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 286.595565ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
12762024-10-22T00:32:21.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 682.347093ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
12772024-10-22T00:32:21.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 728.259427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
12782024-10-22T00:32:21.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 970.117223ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
12792024-10-22T00:32:21.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 497.722643ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
12802024-10-22T00:32:21.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 674.926494ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
12812024-10-22T00:32:21.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.279954101s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
12822024-10-22T00:32:21.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.486962423s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
12832024-10-22T00:32:21.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.111856513s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
12842024-10-22T00:32:21.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.100290524s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
12852024-10-22T00:32:21.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 587.849435ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
12862024-10-22T00:32:21.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.217403583s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
12872024-10-22T00:32:21.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 947.128928ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
12882024-10-22T00:32:22.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 451.751986ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
12892024-10-22T00:32:22.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 987.127561ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
12902024-10-22T00:32:22.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.079068887s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
12912024-10-22T00:32:22.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.047418424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
12922024-10-22T00:32:22.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 899.991079ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
12932024-10-22T00:32:22.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.220669992s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
12942024-10-22T00:32:22.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 700.243488ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
12952024-10-22T00:32:22.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 904.856493ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
12962024-10-22T00:32:22.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 737.449607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
12972024-10-22T00:32:22.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 661.461319ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
12982024-10-22T00:32:22.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.148488488s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
12992024-10-22T00:32:22.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.255895448s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
13002024-10-22T00:32:22.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 779.460774ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
13012024-10-22T00:32:22.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.368534024s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
13022024-10-22T00:32:22.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 791.504994ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
13032024-10-22T00:32:22.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.098976259s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
13042024-10-22T00:32:22.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 738.305736ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
13052024-10-22T00:32:22.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 909.266364ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
13062024-10-22T00:32:22.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 931.120864ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
13072024-10-22T00:32:22.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.222067953s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
13082024-10-22T00:32:23.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.140314703s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
13092024-10-22T00:32:23.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.004288326s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
13102024-10-22T00:32:23.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.39665395s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
13112024-10-22T00:32:23.078ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 544.402319ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
13122024-10-22T00:32:23.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 628.213673ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
13132024-10-22T00:32:23.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.459367714s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
13142024-10-22T00:32:23.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.414215183s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
13152024-10-22T00:32:23.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.327240867s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
13162024-10-22T00:32:23.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 1.424716497s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
13172024-10-22T00:32:23.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 936.252411ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
13182024-10-22T00:32:23.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 659.672898ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
13192024-10-22T00:32:23.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 899.988178ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
13202024-10-22T00:32:23.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.060140807s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
13212024-10-22T00:32:23.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.42669754s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
13222024-10-22T00:32:23.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.317375221s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
13232024-10-22T00:32:23.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 614.363962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
13242024-10-22T00:32:23.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 941.565129ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
13252024-10-22T00:32:23.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 971.425526ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
13262024-10-22T00:32:23.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 871.540119ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
13272024-10-22T00:32:23.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 733.451562ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
13282024-10-22T00:32:24.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 626.820251ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
13292024-10-22T00:32:24.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 911.305901ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
13302024-10-22T00:32:24.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.216653905s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
13312024-10-22T00:32:24.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.286832052s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
13322024-10-22T00:32:24.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 769.302488ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
13332024-10-22T00:32:24.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 926.15793ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
13342024-10-22T00:32:24.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.45331455s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
13352024-10-22T00:32:24.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.266168582s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
13362024-10-22T00:32:24.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 657.263941ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
13372024-10-22T00:32:24.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.020769234s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
13382024-10-22T00:32:24.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 511.868398ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
13392024-10-22T00:32:24.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.373853113s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
13402024-10-22T00:32:24.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.238846337s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
13412024-10-22T00:32:24.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 516.533549ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
13422024-10-22T00:32:24.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 646.691704ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
13432024-10-22T00:32:24.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 784.111488ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
13442024-10-22T00:32:24.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.209251798s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
13452024-10-22T00:32:24.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.301300513s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
13462024-10-22T00:32:25.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.202036406s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
13472024-10-22T00:32:25.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.152412103s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
13482024-10-22T00:32:25.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.123045062s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
13492024-10-22T00:32:25.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.150772894s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
13502024-10-22T00:32:25.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 557.791975ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
13512024-10-22T00:32:25.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 959.880306ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
13522024-10-22T00:32:25.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.246298699s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
13532024-10-22T00:32:25.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.257201675s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
13542024-10-22T00:32:25.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 743.644197ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
13552024-10-22T00:32:25.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 959.973355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
13562024-10-22T00:32:25.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 669.164773ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
13572024-10-22T00:32:25.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 624.700776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
13582024-10-22T00:32:25.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 517.332259ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
13592024-10-22T00:32:25.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 836.00137ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
13602024-10-22T00:32:25.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 692.093072ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
13612024-10-22T00:32:25.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.132473574s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
13622024-10-22T00:32:25.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.336906831s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
13632024-10-22T00:32:26.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 631.27344ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
13642024-10-22T00:32:26.049ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 510.716647ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
13652024-10-22T00:32:26.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 1.025774987s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
13662024-10-22T00:32:26.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 525.24921ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
13672024-10-22T00:32:26.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 696.580608ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
13682024-10-22T00:32:26.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.055841747s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
13692024-10-22T00:32:26.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.339973185s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
13702024-10-22T00:32:26.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 649.007235ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
13712024-10-22T00:32:26.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 653.085007ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
13722024-10-22T00:32:26.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 897.323598ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
13732024-10-22T00:32:26.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 747.144128ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
13742024-10-22T00:32:26.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 907.196559ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
13752024-10-22T00:32:26.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 918.626356ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
13762024-10-22T00:32:26.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 761.127421ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
13772024-10-22T00:32:26.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.127736369s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
13782024-10-22T00:32:26.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.160575744s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
13792024-10-22T00:32:26.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.36540689s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
13802024-10-22T00:32:26.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 612.543971ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
13812024-10-22T00:32:26.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.380815323s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
13822024-10-22T00:32:26.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.144426949s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
13832024-10-22T00:32:26.840ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.148254846s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
13842024-10-22T00:32:26.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.27544619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
13852024-10-22T00:32:26.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 642.748845ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
13862024-10-22T00:32:26.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.049949055s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
13872024-10-22T00:32:27.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 786.144659ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
13882024-10-22T00:32:27.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 506.426637ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
13892024-10-22T00:32:27.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 787.707043ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
13902024-10-22T00:32:27.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.336248326s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
13912024-10-22T00:32:27.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.078681495s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
13922024-10-22T00:32:27.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 830.852067ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
13932024-10-22T00:32:27.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 758.632503ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
13942024-10-22T00:32:27.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 731.012693ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
13952024-10-22T00:32:27.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 715.109119ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
13962024-10-22T00:32:27.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 653.714311ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
13972024-10-22T00:32:27.512ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34579
13982024-10-22T00:32:27.514ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 512 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34579 req_id = 4f3655a6-62e0-46c7-8bec-4f257f64d96f response_code = 200 uri = /baseboard
13992024-10-22T00:32:27.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 803.593969ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
14002024-10-22T00:32:27.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.219741011s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
14012024-10-22T00:32:27.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 607.543888ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
14022024-10-22T00:32:27.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.339109737s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
14032024-10-22T00:32:27.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.035771904s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
14042024-10-22T00:32:27.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.395988054s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
14052024-10-22T00:32:27.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 619.636495ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
14062024-10-22T00:32:27.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 921.700572ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
14072024-10-22T00:32:27.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 960.348701ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
14082024-10-22T00:32:27.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 670.724432ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
14092024-10-22T00:32:28.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 797.004198ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
14102024-10-22T00:32:28.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 718.325244ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
14112024-10-22T00:32:28.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 549.201855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
14122024-10-22T00:32:28.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.158392811s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
14132024-10-22T00:32:28.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 984.143559ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
14142024-10-22T00:32:28.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.367097781s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
14152024-10-22T00:32:28.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.398531214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
14162024-10-22T00:32:28.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.325407265s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
14172024-10-22T00:32:28.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 528.663137ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
14182024-10-22T00:32:28.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.026849633s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
14192024-10-22T00:32:28.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.494990796s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
14202024-10-22T00:32:28.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.276100016s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
14212024-10-22T00:32:28.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 598.058039ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
14222024-10-22T00:32:28.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 1.379598757s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
14232024-10-22T00:32:28.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 703.446678ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
14242024-10-22T00:32:28.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.384571772s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
14252024-10-22T00:32:28.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.138432481s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
14262024-10-22T00:32:28.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 681.175049ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
14272024-10-22T00:32:28.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.331281143s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
14282024-10-22T00:32:28.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.464463443s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
14292024-10-22T00:32:28.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.457729782s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
14302024-10-22T00:32:28.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 853.289059ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
14312024-10-22T00:32:29.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 607.94317ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
14322024-10-22T00:32:29.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.048125901s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
14332024-10-22T00:32:29.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.371272608s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
14342024-10-22T00:32:29.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 537.466825ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
14352024-10-22T00:32:29.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 771.669602ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
14362024-10-22T00:32:29.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 970.23743ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
14372024-10-22T00:32:29.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.370239612s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
14382024-10-22T00:32:29.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.487935741s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
14392024-10-22T00:32:29.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 822.772625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
14402024-10-22T00:32:29.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.170920639s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
14412024-10-22T00:32:29.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.059160043s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
14422024-10-22T00:32:29.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.204745189s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
14432024-10-22T00:32:29.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.455298146s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
14442024-10-22T00:32:29.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.245787047s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
14452024-10-22T00:32:29.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 655.2548ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
14462024-10-22T00:32:30.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.27311596s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
14472024-10-22T00:32:30.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 863.04514ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
14482024-10-22T00:32:30.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.007293977s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
14492024-10-22T00:32:30.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.231819876s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
14502024-10-22T00:32:30.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 514.181179ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
14512024-10-22T00:32:30.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 621.948824ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
14522024-10-22T00:32:30.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.019293987s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
14532024-10-22T00:32:30.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 780.438807ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
14542024-10-22T00:32:30.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.1863997s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
14552024-10-22T00:32:30.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 671.642824ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
14562024-10-22T00:32:30.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 544.504401ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
14572024-10-22T00:32:30.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 551.257199ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
14582024-10-22T00:32:30.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.002600112s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
14592024-10-22T00:32:30.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 725.426986ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
14602024-10-22T00:32:30.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 960.72015ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
14612024-10-22T00:32:30.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 922.151506ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
14622024-10-22T00:32:30.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.385641868s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
14632024-10-22T00:32:30.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 581.934098ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
14642024-10-22T00:32:31.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 987.615437ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
14652024-10-22T00:32:31.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 919.186688ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
14662024-10-22T00:32:31.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.464877142s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
14672024-10-22T00:32:31.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.42944498s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
14682024-10-22T00:32:31.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 888.847298ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
14692024-10-22T00:32:31.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 979.64947ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
14702024-10-22T00:32:31.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.277123362s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
14712024-10-22T00:32:31.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.07347476s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
14722024-10-22T00:32:31.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 987.883244ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
14732024-10-22T00:32:31.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 994.446065ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
14742024-10-22T00:32:31.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 870.399426ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
14752024-10-22T00:32:31.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 975.461649ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
14762024-10-22T00:32:31.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 887.729214ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
14772024-10-22T00:32:31.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 535.325042ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
14782024-10-22T00:32:31.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.33190095s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
14792024-10-22T00:32:31.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.149336383s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
14802024-10-22T00:32:31.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.130667563s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
14812024-10-22T00:32:31.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.275843009s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
14822024-10-22T00:32:31.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 984.464089ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
14832024-10-22T00:32:32.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.066707152s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
14842024-10-22T00:32:32.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 1.292005477s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
14852024-10-22T00:32:32.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 1.297237975s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
14862024-10-22T00:32:32.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 859.21822ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
14872024-10-22T00:32:32.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.347120212s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
14882024-10-22T00:32:32.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 947.623667ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
14892024-10-22T00:32:32.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 1.458333394s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
14902024-10-22T00:32:32.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 702.100901ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
14912024-10-22T00:32:32.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 931.064882ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
14922024-10-22T00:32:32.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 606.476948ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
14932024-10-22T00:32:32.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 1.269952082s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
14942024-10-22T00:32:32.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 918.332843ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
14952024-10-22T00:32:32.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.469996705s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
14962024-10-22T00:32:32.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 763.587641ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
14972024-10-22T00:32:32.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 506.814262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
14982024-10-22T00:32:32.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.128053256s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
14992024-10-22T00:32:32.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 808.778514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
15002024-10-22T00:32:32.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.1530744s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
15012024-10-22T00:32:33.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 858.311574ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
15022024-10-22T00:32:33.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 921.228381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
15032024-10-22T00:32:33.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 1.333495603s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
15042024-10-22T00:32:33.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 703.518165ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
15052024-10-22T00:32:33.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 966.572622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
15062024-10-22T00:32:33.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 660.026469ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
15072024-10-22T00:32:33.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.451144733s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
15082024-10-22T00:32:33.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 992.542987ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
15092024-10-22T00:32:33.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 808.298214ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
15102024-10-22T00:32:33.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 897.555867ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
15112024-10-22T00:32:33.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.328555848s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
15122024-10-22T00:32:33.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 821.106431ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
15132024-10-22T00:32:33.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.459607467s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
15142024-10-22T00:32:33.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 717.163726ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
15152024-10-22T00:32:33.770ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 787.686214ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
15162024-10-22T00:32:33.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 647.452469ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
15172024-10-22T00:32:33.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 635.603569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
15182024-10-22T00:32:33.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.245712002s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
15192024-10-22T00:32:33.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.318142049s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
15202024-10-22T00:32:33.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.355917025s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
15212024-10-22T00:32:34.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 673.254837ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
15222024-10-22T00:32:34.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 1.032446113s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
15232024-10-22T00:32:34.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 670.188821ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
15242024-10-22T00:32:34.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.367224491s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
15252024-10-22T00:32:34.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 1.400687355s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
15262024-10-22T00:32:34.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.163388313s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
15272024-10-22T00:32:34.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 698.26791ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
15282024-10-22T00:32:34.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 932.034542ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
15292024-10-22T00:32:34.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 858.600411ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
15302024-10-22T00:32:34.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 569.147235ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
15312024-10-22T00:32:34.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 590.886306ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
15322024-10-22T00:32:34.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 592.710921ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
15332024-10-22T00:32:34.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 966.299993ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
15342024-10-22T00:32:34.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320") failed: Property not found. retry in 1.405629204s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
15352024-10-22T00:32:34.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 834.740594ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
15362024-10-22T00:32:34.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6") failed: Property not found. retry in 1.483315146s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
15372024-10-22T00:32:34.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 782.458795ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
15382024-10-22T00:32:35.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06") failed: Property not found. retry in 1.435785183s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
15392024-10-22T00:32:35.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696") failed: Property not found. retry in 1.212613823s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
15402024-10-22T00:32:35.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430") failed: Property not found. retry in 943.907594ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
15412024-10-22T00:32:35.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62") failed: Property not found. retry in 1.471303524s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
15422024-10-22T00:32:35.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88") failed: Property not found. retry in 854.52879ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
15432024-10-22T00:32:35.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e") failed: Property not found. retry in 1.262337601s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
15442024-10-22T00:32:35.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f") failed: Property not found. retry in 1.312432431s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
15452024-10-22T00:32:35.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb") failed: Property not found. retry in 1.394524324s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
15462024-10-22T00:32:35.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b") failed: Property not found. retry in 798.45549ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
15472024-10-22T00:32:35.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371") failed: Property not found. retry in 788.088595ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
15482024-10-22T00:32:35.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c") failed: Property not found. retry in 1.040278272s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
15492024-10-22T00:32:35.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e") failed: Property not found. retry in 1.239319667s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
15502024-10-22T00:32:35.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8") failed: Property not found. retry in 1.341694202s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
15512024-10-22T00:32:35.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f") failed: Property not found. retry in 560.256076ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
15522024-10-22T00:32:35.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7") failed: Property not found. retry in 1.367699597s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
15532024-10-22T00:32:35.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049") failed: Property not found. retry in 1.355533815s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
15542024-10-22T00:32:35.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb") failed: Property not found. retry in 525.353878ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
15552024-10-22T00:32:36.978ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_nexus_1ddbaee3-3dbf-4414-b1d3-075d046a666f
15562024-10-22T00:32:36.978ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_df5e8e82-8c42-470c-8f2b-eaaa9abaa7a6
15572024-10-22T00:32:36.978ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_external_dns_1ed57c3f-d967-4545-97bd-c26e5c30bbbb
15582024-10-22T00:32:36.978ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_oximeter_f7ab276a-527a-42f6-b169-e0079c024430
15592024-10-22T00:32:36.978ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_14bbe5e4-bb66-4b10-abdd-57d4f73f950e
15602024-10-22T00:32:36.978ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_pantry_ef33f36a-2317-4f47-821d-4cfbccd3056c
15612024-10-22T00:32:36.978ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_pantry_b6fabd16-0c44-4d4e-8891-220660ced37f
15622024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_bc227b7a-504b-4d14-a0b2-298387ecf3d7
15632024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_cc9c640a-d082-4992-ae26-2265af436049
15642024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_pantry_26cb4910-66e0-447b-a9fe-b56f1eccd33e
15652024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_1a6ad6ee-052d-4f07-aa87-7e5a1e4daa62
15662024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_e25dacf9-31af-4674-910b-dafdcdcd38d8
15672024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_nexus_ac50d2b2-d7d9-4d55-9c28-567f3362fbbb
15682024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_external_dns_e8c37ad4-fd39-4460-b5dc-d6c95ee10320
15692024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_960438c0-57fe-4406-a0be-4f4ee6b84a88
15702024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_1b0d34b4-83ba-4bc1-974b-4b8176749e06
15712024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_clickhouse_c4e4554d-3767-4139-9cd6-7b8f7bc20696
15722024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_crucible_480888f2-5a14-4872-84ee-4203b8e8ad3b
15732024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3325 zone = oxz_nexus_b360f3f1-c2e5-43e4-b254-fe92a717e371
15742024-10-22T00:32:36.979ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15752024-10-22T00:32:36.993ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15762024-10-22T00:32:37.003ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 41607063 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42112 req_id = c2c1adf1-562e-4203-922e-a99adea2a15b response_code = 204 uri = /omicron-zones
15772024-10-22T00:32:37.021ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1321
15782024-10-22T00:32:37.021ZINFOSledAgent (RSS): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
15792024-10-22T00:32:37.022ZINFOSledAgent (RSS): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
15802024-10-22T00:32:37.575ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
15812024-10-22T00:32:37.586ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:719
15822024-10-22T00:32:37.623ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, rib_priority: None, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:886
15832024-10-22T00:32:37.629ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15842024-10-22T00:32:37.748ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15852024-10-22T00:32:37.906ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15862024-10-22T00:32:37.944ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 6.770637395s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
15872024-10-22T00:32:38.100ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15882024-10-22T00:32:38.246ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15892024-10-22T00:32:38.399ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15902024-10-22T00:32:38.772ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15912024-10-22T00:32:39.190ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15922024-10-22T00:32:39.571ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15932024-10-22T00:32:39.850ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15942024-10-22T00:32:40.493ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15952024-10-22T00:32:41.037ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15962024-10-22T00:32:41.526ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15972024-10-22T00:32:42.265ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15982024-10-22T00:32:42.807ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15992024-10-22T00:32:44.239ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
16002024-10-22T00:32:44.753ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.712822554s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
16012024-10-22T00:32:46.041ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
16022024-10-22T00:32:47.351ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
16032024-10-22T00:32:48.573ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/26fc8a25-8c7c-49c9-be24-a8461466dfe6/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
16042024-10-22T00:32:52.187ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "4e8e5c22-7b42-4b8d-8045-f81cd47945e3", "content-length": "173", "date": "Tue, 22 Oct 2024 00:32:52 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"e50e0974-d68b-427b-915b-5bd9a655288a\"", request_id: "4e8e5c22-7b42-4b8d-8045-f81cd47945e3" } file = sled-agent/src/rack_setup/service.rs:948
16052024-10-22T00:32:56.136ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "16f33889-a4ec-4656-9913-09345f2179dd", "content-length": "173", "date": "Tue, 22 Oct 2024 00:32:55 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"e50e0974-d68b-427b-915b-5bd9a655288a\"", request_id: "16f33889-a4ec-4656-9913-09345f2179dd" } file = sled-agent/src/rack_setup/service.rs:948
16062024-10-22T00:32:57.549ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:59515
16072024-10-22T00:32:57.549ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 30 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:59515 req_id = c744e5e4-2fcd-4738-9d0a-2ae45e367086 response_code = 200 uri = /baseboard
16082024-10-22T00:32:59.189ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "a57cc6bc-4d6e-4021-a529-b0c73e93eaea", "content-length": "173", "date": "Tue, 22 Oct 2024 00:32:58 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"e50e0974-d68b-427b-915b-5bd9a655288a\"", request_id: "a57cc6bc-4d6e-4021-a529-b0c73e93eaea" } file = sled-agent/src/rack_setup/service.rs:948
16092024-10-22T00:33:05.626ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "66c12ed6-e0c5-45b7-87b9-a9bd19712e29", "content-length": "173", "date": "Tue, 22 Oct 2024 00:33:05 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"e50e0974-d68b-427b-915b-5bd9a655288a\"", request_id: "66c12ed6-e0c5-45b7-87b9-a9bd19712e29" } file = sled-agent/src/rack_setup/service.rs:948
16102024-10-22T00:33:12.049ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "0f8fe944-62f1-4767-bbe9-6e276f07c9bb", "content-length": "173", "date": "Tue, 22 Oct 2024 00:33:12 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"e50e0974-d68b-427b-915b-5bd9a655288a\"", request_id: "0f8fe944-62f1-4767-bbe9-6e276f07c9bb" } file = sled-agent/src/rack_setup/service.rs:948
16112024-10-22T00:33:16.640ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "70463e1c-2f02-4154-bcfa-301114e09f1c", "content-length": "173", "date": "Tue, 22 Oct 2024 00:33:16 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"e50e0974-d68b-427b-915b-5bd9a655288a\"", request_id: "70463e1c-2f02-4154-bcfa-301114e09f1c" } file = sled-agent/src/rack_setup/service.rs:948
16122024-10-22T00:33:18.944ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:500 new_generation = 1 old_generation = 1 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
16132024-10-22T00:33:19.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53968
16142024-10-22T00:33:19.049ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16152024-10-22T00:33:19.049ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16162024-10-22T00:33:19.049ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-zones.json file = common/src/ledger.rs:177
16172024-10-22T00:33:19.051ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-zones.json file = common/src/ledger.rs:177
16182024-10-22T00:33:19.242ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 194105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53968 req_id = 8e7cc8d4-a19a-4ff0-8ff4-3c410f5436e9 response_code = 200 uri = /inventory
16192024-10-22T00:33:21.085ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35815
16202024-10-22T00:33:21.087ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1978 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35815 req_id = 776173df-c6f2-4ba2-b12f-741b425aaf40 response_code = 200 uri = /vpc-routes
16212024-10-22T00:33:21.087ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40805
16222024-10-22T00:33:21.087ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40805 req_id = 4f4c06e4-32cf-471f-86bf-eacbf9a99b36 response_code = 200 uri = /vpc-routes
16232024-10-22T00:33:21.144ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
16242024-10-22T00:33:21.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 696 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35815 req_id = 4f87ce88-0621-4645-b510-55cbffbd8b02 response_code = 204 uri = /eip-gateways
16252024-10-22T00:33:21.148ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
16262024-10-22T00:33:21.148ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40805 req_id = e239237d-5ed0-45b7-a5cc-fca67cd9f749 response_code = 204 uri = /eip-gateways
16272024-10-22T00:33:21.223ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36247
16282024-10-22T00:33:21.223ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 612 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36247 req_id = f0097279-b8f1-4ab0-a391-623fe48daabc response_code = 200 uri = /v2p
16292024-10-22T00:33:21.249ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50346
16302024-10-22T00:33:21.249ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50346 req_id = 1fd9b243-7d54-47fc-ba0a-046f4eadeb5e response_code = 200 uri = /v2p
16312024-10-22T00:33:21.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3476 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35815 req_id = 8041ac61-32f7-45bc-ad38-91347c17163f response_code = 204 uri = /vpc-routes
16322024-10-22T00:33:21.321ZINFOSledAgent (PortManager): skipping delta compute for subnet file = illumos-utils/src/opte/port_manager.rs:469 new_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 5 } old_vers = RouterVersion { router_id: 001de000-074c-4000-8000-000000000001, version: 5 } subnet = RouterId { vni: Vni(100), kind: System }
16332024-10-22T00:33:21.321ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40805 req_id = 5b9e4247-0343-4275-a3e7-9aa92158644f response_code = 204 uri = /vpc-routes
16342024-10-22T00:33:21.380ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61566
16352024-10-22T00:33:21.380ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 25 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61566 req_id = d70c945c-d357-43de-9d0e-a4cf9efffbfa response_code = 200 uri = /vpc-routes
16362024-10-22T00:33:21.423ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42719
16372024-10-22T00:33:21.424ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42719 req_id = 8f2ea9da-ec79-4999-a0b1-5f22309ac7fa response_code = 200 uri = /v2p
16382024-10-22T00:33:21.460ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
16392024-10-22T00:33:21.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61566 req_id = 5c5a7eef-371f-4f8c-a6c7-4594aa6417c0 response_code = 204 uri = /eip-gateways
16402024-10-22T00:33:21.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61566 req_id = 2da19774-0de8-4882-9805-7fbe5d69febf response_code = 204 uri = /vpc-routes
16412024-10-22T00:33:24.630ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:54049 remote_addr = [fd00:1122:3344:101::d]:35322 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
16422024-10-22T00:33:24.668ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 36892 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = 238eab98-6a12-475a-8adf-5781b1dc0b3b response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
16432024-10-22T00:33:27.581ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40689
16442024-10-22T00:33:27.582ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 62 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40689 req_id = 4428c439-9665-4b1a-8e30-c0716fa11452 response_code = 200 uri = /baseboard
16452024-10-22T00:33:27.622ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33262
16462024-10-22T00:33:27.623ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a9617e4-b230-4cf3-bf60-222f1daf4df2, kind: Service { id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320 }, name: Name("external-dns-e8c37ad4-fd39-4460-b5dc-d6c95ee10320"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 220])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 68a0f950-6467-48bb-b198-77a5253fae12, kind: Service { id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb }, name: Name("external-dns-1ed57c3f-d967-4545-97bd-c26e5c30bbbb"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 169])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60b8c23e-4f5b-4259-9ee0-0845046b29a4, kind: Service { id: ac50d2b2-d7d9-4d55-9c28-567f3362fbbb }, name: Name("nexus-ac50d2b2-d7d9-4d55-9c28-567f3362fbbb"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 115])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 73b1ae2e-5e7d-4c5e-8af0-31d8145c570a, kind: Service { id: b360f3f1-c2e5-43e4-b254-fe92a717e371 }, name: Name("nexus-b360f3f1-c2e5-43e4-b254-fe92a717e371"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 21])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ec446109-19ba-4097-9ed2-643d63373744, kind: Service { id: 1ddbaee3-3dbf-4414-b1d3-075d046a666f }, name: Name("nexus-1ddbaee3-3dbf-4414-b1d3-075d046a666f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 183])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16472024-10-22T00:33:27.623ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16482024-10-22T00:33:27.623ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16492024-10-22T00:33:27.623ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16502024-10-22T00:33:27.623ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16512024-10-22T00:33:27.623ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
16522024-10-22T00:33:27.623ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16532024-10-22T00:33:27.623ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33262 req_id = 864279da-2e57-4228-818e-8dbc91ada44d response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16542024-10-22T00:33:27.625ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:958
16552024-10-22T00:33:27.797ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47529
16562024-10-22T00:33:27.797ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16572024-10-22T00:33:27.797ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16582024-10-22T00:33:27.797ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/config/omicron-zones.json file = common/src/ledger.rs:177
16592024-10-22T00:33:27.797ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/config/omicron-zones.json file = common/src/ledger.rs:177
16602024-10-22T00:33:27.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 198489 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47529 req_id = 7b129fa2-56dc-4464-9c10-2602886c207e response_code = 200 uri = /inventory
16612024-10-22T00:33:29.318ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49145
16622024-10-22T00:33:29.319ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a9617e4-b230-4cf3-bf60-222f1daf4df2, kind: Service { id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320 }, name: Name("external-dns-e8c37ad4-fd39-4460-b5dc-d6c95ee10320"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 220])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 68a0f950-6467-48bb-b198-77a5253fae12, kind: Service { id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb }, name: Name("external-dns-1ed57c3f-d967-4545-97bd-c26e5c30bbbb"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 169])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60b8c23e-4f5b-4259-9ee0-0845046b29a4, kind: Service { id: ac50d2b2-d7d9-4d55-9c28-567f3362fbbb }, name: Name("nexus-ac50d2b2-d7d9-4d55-9c28-567f3362fbbb"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 115])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 73b1ae2e-5e7d-4c5e-8af0-31d8145c570a, kind: Service { id: b360f3f1-c2e5-43e4-b254-fe92a717e371 }, name: Name("nexus-b360f3f1-c2e5-43e4-b254-fe92a717e371"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 21])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ec446109-19ba-4097-9ed2-643d63373744, kind: Service { id: 1ddbaee3-3dbf-4414-b1d3-075d046a666f }, name: Name("nexus-1ddbaee3-3dbf-4414-b1d3-075d046a666f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 183])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16632024-10-22T00:33:29.319ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16642024-10-22T00:33:29.319ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16652024-10-22T00:33:29.319ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16662024-10-22T00:33:29.319ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16672024-10-22T00:33:29.319ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
16682024-10-22T00:33:29.319ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16692024-10-22T00:33:29.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 348 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49145 req_id = d00fbd57-a0bd-4b10-ae6b-1bce5118ffce response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16702024-10-22T00:33:29.375ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50746
16712024-10-22T00:33:29.375ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a9617e4-b230-4cf3-bf60-222f1daf4df2, kind: Service { id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320 }, name: Name("external-dns-e8c37ad4-fd39-4460-b5dc-d6c95ee10320"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 220])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 68a0f950-6467-48bb-b198-77a5253fae12, kind: Service { id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb }, name: Name("external-dns-1ed57c3f-d967-4545-97bd-c26e5c30bbbb"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 169])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60b8c23e-4f5b-4259-9ee0-0845046b29a4, kind: Service { id: ac50d2b2-d7d9-4d55-9c28-567f3362fbbb }, name: Name("nexus-ac50d2b2-d7d9-4d55-9c28-567f3362fbbb"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 115])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 73b1ae2e-5e7d-4c5e-8af0-31d8145c570a, kind: Service { id: b360f3f1-c2e5-43e4-b254-fe92a717e371 }, name: Name("nexus-b360f3f1-c2e5-43e4-b254-fe92a717e371"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 21])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ec446109-19ba-4097-9ed2-643d63373744, kind: Service { id: 1ddbaee3-3dbf-4414-b1d3-075d046a666f }, name: Name("nexus-1ddbaee3-3dbf-4414-b1d3-075d046a666f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 183])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16722024-10-22T00:33:29.375ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16732024-10-22T00:33:29.375ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16742024-10-22T00:33:29.375ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16752024-10-22T00:33:29.375ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16762024-10-22T00:33:29.375ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
16772024-10-22T00:33:29.375ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16782024-10-22T00:33:29.375ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 257 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50746 req_id = 0797d83e-c3af-41d4-a51b-46733fe42701 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16792024-10-22T00:33:29.541ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52484
16802024-10-22T00:33:29.541ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a9617e4-b230-4cf3-bf60-222f1daf4df2, kind: Service { id: e8c37ad4-fd39-4460-b5dc-d6c95ee10320 }, name: Name("external-dns-e8c37ad4-fd39-4460-b5dc-d6c95ee10320"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 186, 220])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 68a0f950-6467-48bb-b198-77a5253fae12, kind: Service { id: 1ed57c3f-d967-4545-97bd-c26e5c30bbbb }, name: Name("external-dns-1ed57c3f-d967-4545-97bd-c26e5c30bbbb"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 182, 169])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 60b8c23e-4f5b-4259-9ee0-0845046b29a4, kind: Service { id: ac50d2b2-d7d9-4d55-9c28-567f3362fbbb }, name: Name("nexus-ac50d2b2-d7d9-4d55-9c28-567f3362fbbb"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 115])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 73b1ae2e-5e7d-4c5e-8af0-31d8145c570a, kind: Service { id: b360f3f1-c2e5-43e4-b254-fe92a717e371 }, name: Name("nexus-b360f3f1-c2e5-43e4-b254-fe92a717e371"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 21])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ec446109-19ba-4097-9ed2-643d63373744, kind: Service { id: 1ddbaee3-3dbf-4414-b1d3-075d046a666f }, name: Name("nexus-1ddbaee3-3dbf-4414-b1d3-075d046a666f"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 183])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
16812024-10-22T00:33:29.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16822024-10-22T00:33:29.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16832024-10-22T00:33:29.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16842024-10-22T00:33:29.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16852024-10-22T00:33:29.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
16862024-10-22T00:33:29.541ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16872024-10-22T00:33:29.541ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 354 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52484 req_id = 9180a25c-34e5-4468-b354-6d01cf683ad7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16882024-10-22T00:33:34.220ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37405
16892024-10-22T00:33:34.220ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37405 req_id = d4492ed3-9147-4ced-8973-7994c4e0890e response_code = 200 uri = /vpc-routes
16902024-10-22T00:33:34.233ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
16912024-10-22T00:33:34.233ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37405 req_id = 190794e9-6c5e-4bfb-98d1-eca8cc70b103 response_code = 204 uri = /eip-gateways
16922024-10-22T00:33:34.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 266 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37405 req_id = 7e19444e-5847-4db3-ac04-53182b1ce5b0 response_code = 204 uri = /vpc-routes
16932024-10-22T00:33:51.263ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51441
16942024-10-22T00:33:51.263ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51441 req_id = e2980de8-93ca-477b-81af-c82f1ebe7ba5 response_code = 200 uri = /vpc-routes
16952024-10-22T00:33:51.283ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
16962024-10-22T00:33:51.283ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51441 req_id = 2c0624b0-07c0-4112-a561-c027dd2cba65 response_code = 204 uri = /eip-gateways
16972024-10-22T00:33:51.292ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60501
16982024-10-22T00:33:51.292ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60501 req_id = 5e9ef7dd-3d52-4443-b3fe-e8388cbac3e2 response_code = 200 uri = /vpc-routes
16992024-10-22T00:33:51.312ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
17002024-10-22T00:33:51.312ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60501 req_id = 316e9178-2842-4d00-97b2-282c10884378 response_code = 204 uri = /eip-gateways
17012024-10-22T00:33:51.323ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 288 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60501 req_id = f8636885-223b-454b-a775-970c63f778bf response_code = 204 uri = /vpc-routes
17022024-10-22T00:33:51.338ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37232
17032024-10-22T00:33:51.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37232 req_id = 2de53c05-ecb4-46f2-afa7-e357b4c28578 response_code = 200 uri = /v2p
17042024-10-22T00:33:51.343ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 198 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51441 req_id = f0084237-71a3-4e91-9191-49a61e65441f response_code = 204 uri = /vpc-routes
17052024-10-22T00:33:51.395ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50311
17062024-10-22T00:33:51.395ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50311 req_id = f390d7ca-670f-49eb-bdc4-3016a1b482f9 response_code = 200 uri = /vpc-routes
17072024-10-22T00:33:51.449ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
17082024-10-22T00:33:51.449ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50311 req_id = d6de55cf-95cf-4b60-af2f-942981e4df79 response_code = 204 uri = /eip-gateways
17092024-10-22T00:33:51.451ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47799
17102024-10-22T00:33:51.452ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17112024-10-22T00:33:51.523ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
17122024-10-22T00:33:51.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145404 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47799 req_id = 31e16069-e51c-4534-b944-1c210d744cca response_code = 204 uri = /switch-ports
17132024-10-22T00:33:51.596ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40897
17142024-10-22T00:33:51.597ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 463 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50311 req_id = 4eac90a1-eff5-40d0-a62c-ab44e76277ae response_code = 204 uri = /vpc-routes
17152024-10-22T00:33:51.597ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62748
17162024-10-22T00:33:51.597ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40897 req_id = 23c42468-e206-496e-ad3d-8b9431851ac4 response_code = 200 uri = /v2p
17172024-10-22T00:33:51.598ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1248 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47799 req_id = 839cb731-de6e-45cf-9d58-1ed2c29e6b53 response_code = 200 uri = /network-bootstore-config
17182024-10-22T00:33:51.598ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17192024-10-22T00:33:51.646ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
17202024-10-22T00:33:51.653ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
17212024-10-22T00:33:51.654ZINFOSledAgent (bootstore): Writing ledger to /pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
17222024-10-22T00:33:51.655ZINFOSledAgent (bootstore): Writing ledger to /pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
17232024-10-22T00:33:51.655ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
17242024-10-22T00:33:51.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4851 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47799 req_id = 8ba2b47f-ba06-40af-ab35-ad7b5bf437c0 response_code = 204 uri = /network-bootstore-config
17252024-10-22T00:33:51.675ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47685
17262024-10-22T00:33:51.676ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 279 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47685 req_id = 9cdb1554-3c49-40ad-98c6-883616900097 response_code = 200 uri = /v2p
17272024-10-22T00:33:51.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47070
17282024-10-22T00:33:51.710ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17292024-10-22T00:33:51.710ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 111942 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62748 req_id = 3c531ac3-b493-4d14-9519-1434eae4ef6b response_code = 204 uri = /switch-ports
17302024-10-22T00:33:51.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 767 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62748 req_id = 95af5acb-36c4-47ae-85bc-05e2b8bea8da response_code = 200 uri = /network-bootstore-config
17312024-10-22T00:33:51.752ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
17322024-10-22T00:33:51.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126147 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:47070 req_id = ea463a80-d5c3-4129-9088-7876714576ea response_code = 204 uri = /switch-ports
17332024-10-22T00:33:51.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47070 req_id = b5895cb8-dc29-49c5-95e8-dff54cb6af37 response_code = 200 uri = /network-bootstore-config
17342024-10-22T00:33:54.639ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8338 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = 44405e1d-0e88-4220-b185-8e223c67a6d9 response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
17352024-10-22T00:33:57.642ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50668
17362024-10-22T00:33:57.642ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 39 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50668 req_id = 0bc7d81e-f3ea-4f0b-9e0f-e1f748967b95 response_code = 200 uri = /baseboard
17372024-10-22T00:34:21.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64792
17382024-10-22T00:34:21.072ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64792 req_id = 19f3d945-1dee-4008-9266-373176cc10b2 response_code = 200 uri = /vpc-routes
17392024-10-22T00:34:21.135ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
17402024-10-22T00:34:21.135ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64792 req_id = eb188a46-01fe-47be-9777-97d9d4767811 response_code = 204 uri = /eip-gateways
17412024-10-22T00:34:21.144ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62235
17422024-10-22T00:34:21.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62235 req_id = 1f642777-1ff0-4bf7-bc9c-4772870f6c7d response_code = 200 uri = /vpc-routes
17432024-10-22T00:34:21.148ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 209 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64792 req_id = 7ece183c-9ea3-41cf-bb46-718a9a956c31 response_code = 204 uri = /vpc-routes
17442024-10-22T00:34:21.227ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60744
17452024-10-22T00:34:21.227ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 195 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60744 req_id = 6a84f72f-0448-4067-9771-42091ba78223 response_code = 200 uri = /v2p
17462024-10-22T00:34:21.238ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
17472024-10-22T00:34:21.238ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62235 req_id = f712d218-d170-4763-bcb7-88af97a2ba9b response_code = 204 uri = /eip-gateways
17482024-10-22T00:34:21.250ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 248 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62235 req_id = e1c23f04-288d-44a0-b139-b81b166d3a0a response_code = 204 uri = /vpc-routes
17492024-10-22T00:34:21.319ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50565
17502024-10-22T00:34:21.319ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17512024-10-22T00:34:21.367ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
17522024-10-22T00:34:21.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121657 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50565 req_id = b1877848-0331-4511-a3d3-ff8b6ea7d1ec response_code = 204 uri = /switch-ports
17532024-10-22T00:34:21.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50190
17542024-10-22T00:34:21.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53517
17552024-10-22T00:34:21.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49184
17562024-10-22T00:34:21.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 304 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50565 req_id = 909498b5-3c0a-4587-83ec-fdac0167ae8d response_code = 200 uri = /network-bootstore-config
17572024-10-22T00:34:21.442ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17582024-10-22T00:34:21.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53517 req_id = f0e0b6ec-ebb0-4e8f-ae2d-138f8c8ce07d response_code = 200 uri = /vpc-routes
17592024-10-22T00:34:21.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 307 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50190 req_id = 3e03a853-9251-4cb4-b9af-d18a4c335815 response_code = 200 uri = /v2p
17602024-10-22T00:34:21.463ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
17612024-10-22T00:34:21.463ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 759 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53517 req_id = 41e19816-7c65-4e2b-924c-23b6f71b07b8 response_code = 204 uri = /eip-gateways
17622024-10-22T00:34:21.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1023 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53517 req_id = d73996f9-da12-4acd-80fa-7c9fbce843fc response_code = 204 uri = /vpc-routes
17632024-10-22T00:34:21.484ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62504
17642024-10-22T00:34:21.484ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62504 req_id = 1bf1ccc1-4798-4268-9b73-cebe47d33868 response_code = 200 uri = /v2p
17652024-10-22T00:34:21.487ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
17662024-10-22T00:34:21.547ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33557
17672024-10-22T00:34:21.552ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17682024-10-22T00:34:21.552ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110324 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49184 req_id = 02650acc-b957-4405-90f8-8e4ca9f66592 response_code = 204 uri = /switch-ports
17692024-10-22T00:34:21.556ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 788 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49184 req_id = 1b5f6f74-d743-47fc-9753-b444e9c34e48 response_code = 200 uri = /network-bootstore-config
17702024-10-22T00:34:21.595ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
17712024-10-22T00:34:21.657ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110059 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33557 req_id = 2eb77d31-4521-473a-9798-cedbf56d0d92 response_code = 204 uri = /switch-ports
17722024-10-22T00:34:21.657ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33557 req_id = 4dbc67d4-e682-416c-b6f9-7b1c7909d95e response_code = 200 uri = /network-bootstore-config
17732024-10-22T00:34:24.639ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8945 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = bac66d11-4cff-47aa-94fc-1451b4cb6a52 response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
17742024-10-22T00:34:27.675ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47074
17752024-10-22T00:34:27.676ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 62 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47074 req_id = 4c1ea1ae-1ad8-4cbd-bed8-df876d87342d response_code = 200 uri = /baseboard
17762024-10-22T00:34:51.289ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47054
17772024-10-22T00:34:51.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 745 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47054 req_id = ce2a38a6-b393-48e1-9387-bb323cfc4716 response_code = 200 uri = /vpc-routes
17782024-10-22T00:34:51.352ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
17792024-10-22T00:34:51.352ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47054 req_id = 882656d0-4cdc-4082-abdc-cc48d2e154c2 response_code = 204 uri = /eip-gateways
17802024-10-22T00:34:51.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 295 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47054 req_id = d49da878-5b97-4ac7-abc0-8aff65cf1d53 response_code = 204 uri = /vpc-routes
17812024-10-22T00:34:51.394ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60003
17822024-10-22T00:34:51.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60003 req_id = c45824aa-5cc5-4529-b6cd-6a61fe8b094b response_code = 200 uri = /vpc-routes
17832024-10-22T00:34:51.415ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56453
17842024-10-22T00:34:51.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 187 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56453 req_id = 0ffcdc78-6092-41f8-8d6c-e3ba02875f16 response_code = 200 uri = /v2p
17852024-10-22T00:34:51.416ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
17862024-10-22T00:34:51.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60003 req_id = 6a7b794e-fd50-42c3-9187-5e6aba8f0c4b response_code = 204 uri = /eip-gateways
17872024-10-22T00:34:51.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 282 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60003 req_id = 68bf536d-168a-4375-b330-76e7adc88bbf response_code = 204 uri = /vpc-routes
17882024-10-22T00:34:51.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43172
17892024-10-22T00:34:51.486ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17902024-10-22T00:34:51.529ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
17912024-10-22T00:34:51.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99363 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43172 req_id = 79925e98-0511-4dc4-a85d-f7bcf141794f response_code = 204 uri = /switch-ports
17922024-10-22T00:34:51.586ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41965
17932024-10-22T00:34:51.586ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38138
17942024-10-22T00:34:51.586ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 223 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38138 req_id = f5a753fb-2d82-481b-bb1b-381d4237f54f response_code = 200 uri = /v2p
17952024-10-22T00:34:51.586ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
17962024-10-22T00:34:51.586ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 316 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43172 req_id = 7f09e2c0-7646-4c0e-8877-4c45624502b7 response_code = 200 uri = /network-bootstore-config
17972024-10-22T00:34:51.592ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58772
17982024-10-22T00:34:51.593ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58772 req_id = 58f83447-68d3-4046-be4b-5485532c85ec response_code = 200 uri = /v2p
17992024-10-22T00:34:51.593ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62935
18002024-10-22T00:34:51.593ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62935 req_id = 6252a01e-c41f-4b7e-b9a6-78ec98c41933 response_code = 200 uri = /vpc-routes
18012024-10-22T00:34:51.619ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18022024-10-22T00:34:51.636ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18032024-10-22T00:34:51.636ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 734 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62935 req_id = a1f6c9df-e745-46b8-8f5d-b09d16ee9a97 response_code = 204 uri = /eip-gateways
18042024-10-22T00:34:51.650ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 981 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62935 req_id = a9b64e7b-0d9b-43c3-980c-2a85c81ce45d response_code = 204 uri = /vpc-routes
18052024-10-22T00:34:51.651ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39853
18062024-10-22T00:34:51.684ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
18072024-10-22T00:34:51.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97925 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:41965 req_id = ec3bf3ce-401b-43a3-a615-fedca2334031 response_code = 204 uri = /switch-ports
18082024-10-22T00:34:51.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 346 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41965 req_id = 9e03af9b-3719-498d-86fe-d74995afc39c response_code = 200 uri = /network-bootstore-config
18092024-10-22T00:34:51.721ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18102024-10-22T00:34:51.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124653 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39853 req_id = 2ea13f8c-f7ed-424c-9313-1a940b7e38a3 response_code = 204 uri = /switch-ports
18112024-10-22T00:34:51.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39853 req_id = e43e35bd-a1f4-4023-afc6-ade915e39d0b response_code = 200 uri = /network-bootstore-config
18122024-10-22T00:34:54.639ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7824 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = 0039f9fd-6f80-4efd-a0b0-eb94c18ec2b0 response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
18132024-10-22T00:34:57.729ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45626
18142024-10-22T00:34:57.729ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 31 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45626 req_id = 17c559e0-e83b-450c-824d-95b999d4f8b9 response_code = 200 uri = /baseboard
18152024-10-22T00:35:21.126ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43435
18162024-10-22T00:35:21.126ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43435 req_id = 7cda7bc9-fa02-4b51-b963-8ba73e086148 response_code = 200 uri = /vpc-routes
18172024-10-22T00:35:21.145ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60063
18182024-10-22T00:35:21.147ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60063 req_id = dbec394c-5083-4503-a549-91d75033caa9 response_code = 200 uri = /vpc-routes
18192024-10-22T00:35:21.184ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18202024-10-22T00:35:21.184ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43435 req_id = 121b0a5b-b9f6-4e5c-97b2-7c5dda191478 response_code = 204 uri = /eip-gateways
18212024-10-22T00:35:21.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 205 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43435 req_id = fb37be18-ad88-4690-82a0-6e9af629af84 response_code = 204 uri = /vpc-routes
18222024-10-22T00:35:21.205ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35114
18232024-10-22T00:35:21.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 215 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35114 req_id = 0e540c10-ddee-42b1-a753-21ff9331dbd4 response_code = 200 uri = /v2p
18242024-10-22T00:35:21.210ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45330
18252024-10-22T00:35:21.210ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
18262024-10-22T00:35:21.254ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18272024-10-22T00:35:21.314ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103606 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45330 req_id = 45972511-c1bb-4662-9926-84dd5aeab36f response_code = 204 uri = /switch-ports
18282024-10-22T00:35:21.314ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63326
18292024-10-22T00:35:21.314ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18302024-10-22T00:35:21.314ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60063 req_id = c7bb6464-2eea-4274-8d72-e8e0f2c0a765 response_code = 204 uri = /eip-gateways
18312024-10-22T00:35:21.314ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65501
18322024-10-22T00:35:21.314ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 243 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45330 req_id = 05769b7f-f012-4900-a534-e0331414a8e1 response_code = 200 uri = /network-bootstore-config
18332024-10-22T00:35:21.315ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
18342024-10-22T00:35:21.315ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 228 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63326 req_id = ca01e009-d782-4ac3-bca3-e49cf6a5e8b8 response_code = 200 uri = /v2p
18352024-10-22T00:35:21.324ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 298 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60063 req_id = 5d27973f-c8e2-4a25-b7ef-b15b435ca00e response_code = 204 uri = /vpc-routes
18362024-10-22T00:35:21.334ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45538
18372024-10-22T00:35:21.334ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45538 req_id = 5fad4370-39f5-4614-b79d-ceb0abd938c3 response_code = 200 uri = /vpc-routes
18382024-10-22T00:35:21.361ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18392024-10-22T00:35:21.388ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18402024-10-22T00:35:21.388ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:45538 req_id = 03e44774-2a35-40c5-891a-d6205ee081b3 response_code = 204 uri = /eip-gateways
18412024-10-22T00:35:21.400ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 608 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:45538 req_id = 8cb070eb-a47e-4552-b13a-fcd3032f86e0 response_code = 204 uri = /vpc-routes
18422024-10-22T00:35:21.433ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47902
18432024-10-22T00:35:21.443ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
18442024-10-22T00:35:21.443ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 128595 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:65501 req_id = 793f32ec-c6d5-43ff-88d6-d2b40e4df986 response_code = 204 uri = /switch-ports
18452024-10-22T00:35:21.449ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 842 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65501 req_id = 711cf649-4f87-4702-a042-d14b1f2488a9 response_code = 200 uri = /network-bootstore-config
18462024-10-22T00:35:21.496ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18472024-10-22T00:35:21.571ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44653
18482024-10-22T00:35:21.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 217 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44653 req_id = 077ea3c4-3ed7-43f2-803f-b7edb06dbfaa response_code = 200 uri = /v2p
18492024-10-22T00:35:21.578ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145179 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:47902 req_id = 7dd20ccf-a786-4734-a885-3bfc4e5b9b04 response_code = 204 uri = /switch-ports
18502024-10-22T00:35:21.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47902 req_id = 5a990a4f-c050-4c10-a402-6ac89f8e9cc8 response_code = 200 uri = /network-bootstore-config
18512024-10-22T00:35:24.643ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 5705 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = 91ef48c9-d361-47a6-bffa-a930167fc933 response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
18522024-10-22T00:35:27.763ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:33637
18532024-10-22T00:35:27.763ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 33 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:33637 req_id = cfb77313-4bb7-48e8-a1a7-262e620a85da response_code = 200 uri = /baseboard
18542024-10-22T00:35:51.088ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
18552024-10-22T00:35:51.307ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63152
18562024-10-22T00:35:51.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63152 req_id = c48fb644-aa37-446a-83af-15968a50cfdf response_code = 200 uri = /vpc-routes
18572024-10-22T00:35:51.337ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62342
18582024-10-22T00:35:51.337ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62342 req_id = fd724367-1812-4e57-87d8-5fe94e88d3c9 response_code = 200 uri = /vpc-routes
18592024-10-22T00:35:51.403ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18602024-10-22T00:35:51.403ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63152 req_id = 7ce65f25-d12d-4acf-a45c-8847e181bf04 response_code = 204 uri = /eip-gateways
18612024-10-22T00:35:51.414ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18622024-10-22T00:35:51.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62342 req_id = eefc056c-4def-4d69-9e89-f573a86b2d8d response_code = 204 uri = /eip-gateways
18632024-10-22T00:35:51.417ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 386 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63152 req_id = 77976350-d7ae-480f-9496-83b21ee192d0 response_code = 204 uri = /vpc-routes
18642024-10-22T00:35:51.432ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37185
18652024-10-22T00:35:51.432ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37185 req_id = 982d44b7-8c18-4972-b88c-afacff8c1261 response_code = 200 uri = /v2p
18662024-10-22T00:35:51.455ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 230 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62342 req_id = b6c8cdf4-e7b7-4592-a00a-7e317920caf3 response_code = 204 uri = /vpc-routes
18672024-10-22T00:35:51.458ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43796
18682024-10-22T00:35:51.458ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43796 req_id = 2478cf57-cce6-4c2f-8617-77a4404eb379 response_code = 200 uri = /vpc-routes
18692024-10-22T00:35:51.489ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39877
18702024-10-22T00:35:51.489ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
18712024-10-22T00:35:51.540ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18722024-10-22T00:35:51.614ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124637 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39877 req_id = 0d48bbfc-3ab6-4db6-8544-8b43dc856edb response_code = 204 uri = /switch-ports
18732024-10-22T00:35:51.614ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50358
18742024-10-22T00:35:51.614ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18752024-10-22T00:35:51.614ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47086
18762024-10-22T00:35:51.614ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43796 req_id = b7d30d1f-3ecd-4b5b-8549-ebf7ecd036c3 response_code = 204 uri = /eip-gateways
18772024-10-22T00:35:51.614ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 255 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39877 req_id = cfaeb6b9-fa86-4f8d-a8d2-09b58b9ad266 response_code = 200 uri = /network-bootstore-config
18782024-10-22T00:35:51.614ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47086 req_id = 3bf4c4b5-ddc2-44b5-8650-0b0966d108bb response_code = 200 uri = /v2p
18792024-10-22T00:35:51.614ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
18802024-10-22T00:35:51.625ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 399 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43796 req_id = 1d8bf987-e7d0-4aa1-94bc-aa14ddc0802e response_code = 204 uri = /vpc-routes
18812024-10-22T00:35:51.626ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46535
18822024-10-22T00:35:51.626ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46535 req_id = b3f92f9c-9763-445f-916b-1394e7005542 response_code = 200 uri = /v2p
18832024-10-22T00:35:51.653ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18842024-10-22T00:35:51.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38530
18852024-10-22T00:35:51.705ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
18862024-10-22T00:35:51.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91161 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50358 req_id = 3e734a45-0815-448b-9671-65e4b1e29db4 response_code = 204 uri = /switch-ports
18872024-10-22T00:35:51.710ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 702 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50358 req_id = 66bd4230-0da6-42a5-a9ab-41f2a25093d2 response_code = 200 uri = /network-bootstore-config
18882024-10-22T00:35:51.741ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
18892024-10-22T00:35:51.800ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 113428 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38530 req_id = 2ecd29b1-6c74-4061-a2b6-8ebc4680d16e response_code = 204 uri = /switch-ports
18902024-10-22T00:35:51.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38530 req_id = 41c072e7-0e2c-4a0a-b996-2f0e269d8ca1 response_code = 200 uri = /network-bootstore-config
18912024-10-22T00:35:54.690ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7596 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = 13157f10-5300-400c-ab68-a6e607876568 response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
18922024-10-22T00:35:57.797ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:33908
18932024-10-22T00:35:57.798ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 29 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:33908 req_id = d81110ba-a523-48fe-9747-5788760e8523 response_code = 200 uri = /baseboard
18942024-10-22T00:36:21.054ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65287
18952024-10-22T00:36:21.054ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65287 req_id = 954935e3-71c8-4419-893d-47feeb8d1771 response_code = 200 uri = /vpc-routes
18962024-10-22T00:36:21.114ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
18972024-10-22T00:36:21.114ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:65287 req_id = b9696430-04d7-4bb7-bd65-f21172228398 response_code = 204 uri = /eip-gateways
18982024-10-22T00:36:21.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 225 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:65287 req_id = 2713a009-33c3-4231-9386-7e570dace55f response_code = 204 uri = /vpc-routes
18992024-10-22T00:36:21.194ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51305
19002024-10-22T00:36:21.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51305 req_id = 416e2b08-5c38-4bbe-9f57-0467494c3aac response_code = 200 uri = /vpc-routes
19012024-10-22T00:36:21.253ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
19022024-10-22T00:36:21.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51305 req_id = 1b716233-cfa6-4b7b-9825-0789e83cc3d3 response_code = 204 uri = /eip-gateways
19032024-10-22T00:36:21.262ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 195 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51305 req_id = e08dae59-a76b-4396-af18-61559793bc7c response_code = 204 uri = /vpc-routes
19042024-10-22T00:36:21.287ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50191
19052024-10-22T00:36:21.287ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50191 req_id = 25a4fa07-0305-4f7b-8e05-7c2b0161d1e2 response_code = 200 uri = /v2p
19062024-10-22T00:36:21.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37009
19072024-10-22T00:36:21.310ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
19082024-10-22T00:36:21.351ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
19092024-10-22T00:36:21.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96819 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37009 req_id = 08774b0a-9d14-45cd-a9e2-3bb636111045 response_code = 204 uri = /switch-ports
19102024-10-22T00:36:21.407ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45565
19112024-10-22T00:36:21.407ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43086
19122024-10-22T00:36:21.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43086 req_id = 1aab7507-b0af-4d5e-8445-50e875ba6623 response_code = 200 uri = /vpc-routes
19132024-10-22T00:36:21.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37009 req_id = bd770f4d-b5a8-4883-8d3a-cdc141bdaee3 response_code = 200 uri = /network-bootstore-config
19142024-10-22T00:36:21.409ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
19152024-10-22T00:36:21.409ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39750
19162024-10-22T00:36:21.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 347 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39750 req_id = bc16e342-e268-479a-9289-9b4ea31e6bbd response_code = 200 uri = /v2p
19172024-10-22T00:36:21.420ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
19182024-10-22T00:36:21.420ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43086 req_id = 19da6132-05f0-4c1b-b3f1-781bd0b2eca8 response_code = 204 uri = /eip-gateways
19192024-10-22T00:36:21.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 693 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43086 req_id = c09df07a-1b05-4b9e-a293-a6d36bb767d6 response_code = 204 uri = /vpc-routes
19202024-10-22T00:36:21.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42092
19212024-10-22T00:36:21.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 292 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42092 req_id = bba5f083-1b2c-48e8-a759-342b0d0d2c01 response_code = 200 uri = /v2p
19222024-10-22T00:36:21.451ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
19232024-10-22T00:36:21.507ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98506 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45565 req_id = dbb8cf7b-8107-4b37-87ef-a5f34cf1b5bc response_code = 204 uri = /switch-ports
19242024-10-22T00:36:21.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45565 req_id = 234390df-fd06-403e-ba8f-efbd891534e5 response_code = 200 uri = /network-bootstore-config
19252024-10-22T00:36:21.539ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65069
19262024-10-22T00:36:21.539ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
19272024-10-22T00:36:21.577ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
19282024-10-22T00:36:21.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89122 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:65069 req_id = 690ba03f-e6c4-4f6c-aec3-7a5f3de9c585 response_code = 204 uri = /switch-ports
19292024-10-22T00:36:21.629ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65069 req_id = acceb42a-bdbc-4468-b302-7b4cbc3bf3bc response_code = 200 uri = /network-bootstore-config
19302024-10-22T00:36:24.691ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8307 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = b2693800-b27a-4108-86e5-c0f57cdeb610 response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
19312024-10-22T00:36:27.832ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61164
19322024-10-22T00:36:27.832ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 34 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61164 req_id = f703be06-0003-41be-af26-e7be52ef5655 response_code = 200 uri = /baseboard
19332024-10-22T00:36:51.292ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64984
19342024-10-22T00:36:51.293ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64984 req_id = 6fdbfa41-5d7e-452d-a3a6-53b6bdac17d9 response_code = 200 uri = /vpc-routes
19352024-10-22T00:36:51.312ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
19362024-10-22T00:36:51.312ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64984 req_id = eaabff16-636d-4205-b275-370cdf29be40 response_code = 204 uri = /eip-gateways
19372024-10-22T00:36:51.314ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35627
19382024-10-22T00:36:51.314ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35627 req_id = 6ed056c5-cc22-4d6d-90fa-044769b5b10c response_code = 200 uri = /vpc-routes
19392024-10-22T00:36:51.374ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
19402024-10-22T00:36:51.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35627 req_id = 6ba88607-87b3-4446-b41e-d8378e0347dd response_code = 204 uri = /eip-gateways
19412024-10-22T00:36:51.383ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35627 req_id = ad3bdd45-168c-4ed7-a68e-59936c86416d response_code = 204 uri = /vpc-routes
19422024-10-22T00:36:51.437ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41996
19432024-10-22T00:36:51.437ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41996 req_id = 7e7a59fe-5127-4655-8559-eba51ac9e475 response_code = 200 uri = /v2p
19442024-10-22T00:36:51.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64984 req_id = dbd437d9-2a09-4d33-91ab-59c247f0cf05 response_code = 204 uri = /vpc-routes
19452024-10-22T00:36:51.526ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41914
19462024-10-22T00:36:51.526ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41914 req_id = 39646ebb-a62a-409f-963e-d721b0803a06 response_code = 200 uri = /vpc-routes
19472024-10-22T00:36:51.528ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55311
19482024-10-22T00:36:51.528ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
19492024-10-22T00:36:51.577ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
19502024-10-22T00:36:51.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99908 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:55311 req_id = f027da87-3568-4d9b-bc11-c66c98bd5287 response_code = 204 uri = /switch-ports
19512024-10-22T00:36:51.628ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
19522024-10-22T00:36:51.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54859
19532024-10-22T00:36:51.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41914 req_id = 441a62bf-13e2-4d2d-a152-20f13625ea75 response_code = 204 uri = /eip-gateways
19542024-10-22T00:36:51.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49166
19552024-10-22T00:36:51.629ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 273 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55311 req_id = 5923abca-275d-45ce-9e20-ef7c370e6d86 response_code = 200 uri = /network-bootstore-config
19562024-10-22T00:36:51.629ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54859 req_id = c6a5f445-4e19-43c4-ac63-d165d331e2be response_code = 200 uri = /v2p
19572024-10-22T00:36:51.629ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
19582024-10-22T00:36:51.638ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 438 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41914 req_id = 76af31b1-3545-42c9-b4f0-ec667f157e41 response_code = 204 uri = /vpc-routes
19592024-10-22T00:36:51.670ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
19602024-10-22T00:36:51.724ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47657
19612024-10-22T00:36:51.725ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47657 req_id = c9c307b3-aaf7-4ad2-9a7e-4bd30bc24e39 response_code = 200 uri = /v2p
19622024-10-22T00:36:51.725ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96361 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49166 req_id = a6252442-d665-4428-9257-3391f2d23cca response_code = 204 uri = /switch-ports
19632024-10-22T00:36:51.725ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49166 req_id = f2047140-d86a-432d-af26-07618655e4ee response_code = 200 uri = /network-bootstore-config
19642024-10-22T00:36:51.787ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52555
19652024-10-22T00:36:51.788ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
19662024-10-22T00:36:51.821ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
19672024-10-22T00:36:51.873ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 85279 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52555 req_id = 59e881cb-a467-4505-86e0-479615485cf0 response_code = 204 uri = /switch-ports
19682024-10-22T00:36:51.873ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52555 req_id = 83d443cd-871a-4324-bde6-5c6068c4f334 response_code = 200 uri = /network-bootstore-config
19692024-10-22T00:36:54.690ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7485 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = eef6d132-dbfc-448b-994b-a67f79fcd17c response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
19702024-10-22T00:36:57.870ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48811
19712024-10-22T00:36:57.870ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 69 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48811 req_id = 07784bf0-ae76-460a-bcd4-b302af31da04 response_code = 200 uri = /baseboard
19722024-10-22T00:37:13.328ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34055
19732024-10-22T00:37:13.329ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34055 req_id = 385e4fff-2f75-4e77-b01d-1b9978648941 response_code = 200 uri = /v2p
19742024-10-22T00:37:13.329ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:838 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(15647631) }
19752024-10-22T00:37:13.329ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34055 req_id = a5f0307f-aa99-4f36-abad-b3e6cc8730be response_code = 204 uri = /v2p
19762024-10-22T00:37:13.446ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47944
19772024-10-22T00:37:13.451ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:555 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(15647631))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-25f22ea33b9", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 08e5f8d1-e204-4e4d-b8b3-c75329b61e11, read_only_parent: Some(Volume { block_size: 512, id: ad7fe769-0e96-4495-94d9-6e369aa403d2, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0c4754c7-5cf0-4a42-aab7-ea526a8214ed, key: Some("ZDc1ZRrtCCKlSWsuf+tVbMmpA0de7FWPPY5i+OG8vww="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19002", "[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::15]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 08e5f8d1-e204-4e4d-b8b3-c75329b61e11, key: Some("ik5meog9CwG2LjQYksqhhhSjlx1A3OA0tMyqFw9/XRQ="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19000", "[fd00:1122:3344:101::17]:19000", "[fd00:1122:3344:101::12]:19002"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-25f22ea33b9" }] }), cloud_init_bytes: Some(..) } instance_id = 7d023311-eb77-4625-832f-45e1831adeee metadata = InstanceMetadata { silo_id: f6ac13ed-f57a-4014-80a3-c9a5cf418931, project_id: 6c5a4080-dc3f-45f7-a314-92a5612cedf4 } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-22T00:37:13.444107687Z }
19782024-10-22T00:37:13.451ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:588 instance_id = 7d023311-eb77-4625-832f-45e1831adeee migration_id = None propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
19792024-10-22T00:37:13.452ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1139 instance_id = 7d023311-eb77-4625-832f-45e1831adeee migration_id = None propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(15647631))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: bb2562cc-387d-4dec-b6ab-7f8cb3916a5c, kind: Instance { id: 7d023311-eb77-4625-832f-45e1831adeee }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(15647631), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-25f22ea33b9", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 08e5f8d1-e204-4e4d-b8b3-c75329b61e11, read_only_parent: Some(Volume { block_size: 512, id: ad7fe769-0e96-4495-94d9-6e369aa403d2, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0c4754c7-5cf0-4a42-aab7-ea526a8214ed, key: Some("ZDc1ZRrtCCKlSWsuf+tVbMmpA0de7FWPPY5i+OG8vww="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19002", "[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::15]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 08e5f8d1-e204-4e4d-b8b3-c75329b61e11, key: Some("ik5meog9CwG2LjQYksqhhhSjlx1A3OA0tMyqFw9/XRQ="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19000", "[fd00:1122:3344:101::17]:19000", "[fd00:1122:3344:101::12]:19002"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-25f22ea33b9" }] }), cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-22T00:37:13.444107687Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
19802024-10-22T00:37:13.453ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7078 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47944 req_id = 90cfda2f-0439-4056-924c-a7ad71df147e response_code = 200 uri = /vmms/0aef4f71-019e-41bc-a78a-9c7292529cab
19812024-10-22T00:37:13.562ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57235
19822024-10-22T00:37:13.563ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 338 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57235 req_id = fde81edc-b1c6-4e49-8051-ea93916d01e1 response_code = 200 uri = /vmms/0aef4f71-019e-41bc-a78a-9c7292529cab/state
19832024-10-22T00:37:13.564ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 15647631 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
19842024-10-22T00:37:13.564ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
19852024-10-22T00:37:13.603ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab file = illumos-utils/src/zone.rs:316 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
19862024-10-22T00:37:13.625ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab file = illumos-utils/src/zone.rs:354 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
19872024-10-22T00:37:13.683ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46992
19882024-10-22T00:37:13.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46992 req_id = caae506b-c38e-4a74-89b1-883e9416cc22 response_code = 200 uri = /vpc-routes
19892024-10-22T00:37:13.701ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {bb2562cc-387d-4dec-b6ab-7f8cb3916a5c: {10.151.2.175: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}, 10.151.2.176: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}}} }
19902024-10-22T00:37:14.960ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
19912024-10-22T00:37:14.961ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab zone file = illumos-utils/src/running_zone.rs:488 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19922024-10-22T00:37:15.190ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 25.45643ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19932024-10-22T00:37:15.228ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 81.688756ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19942024-10-22T00:37:15.329ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 121.172437ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19952024-10-22T00:37:15.468ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 211.161885ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19962024-10-22T00:37:15.694ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 500.14651ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19972024-10-22T00:37:16.207ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 1.25600762s file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19982024-10-22T00:37:17.473ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 1.039139782s file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
19992024-10-22T00:37:18.522ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 882.246863ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
20002024-10-22T00:37:19.414ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 825.130321ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab zone = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
20012024-10-22T00:37:20.270ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab file = sled-agent/src/instance.rs:1633 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20022024-10-22T00:37:20.279ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 41.751544ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20032024-10-22T00:37:20.330ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 95.514703ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20042024-10-22T00:37:20.435ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab") failed: Property not found. retry in 231.507869ms file = illumos-utils/src/svc.rs:36 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20052024-10-22T00:37:20.677ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1642 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20062024-10-22T00:37:20.714ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1666 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20072024-10-22T00:37:20.854ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:835 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20082024-10-22T00:37:20.856ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1203 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
20092024-10-22T00:37:20.856ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7154374 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46992 req_id = 43bcb347-f1f5-41fc-b7fa-50a3696b7277 response_code = 204 uri = /eip-gateways
20102024-10-22T00:37:20.856ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-10-22T00:37:20.856534066Z } file = sled-agent/src/instance.rs:685 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20112024-10-22T00:37:20.856ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 7d023311-eb77-4625-832f-45e1831adeee new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-22T00:37:20.856534066Z } propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20122024-10-22T00:37:20.856ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-22T00:37:20.856534066Z }, migration_in: None, migration_out: None }
20132024-10-22T00:37:20.888ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-10-22T00:37:20.888113528Z } file = sled-agent/src/instance.rs:685 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20142024-10-22T00:37:20.888ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 7d023311-eb77-4625-832f-45e1831adeee new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-22T00:37:20.888113528Z } propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20152024-10-22T00:37:20.888ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-22T00:37:20.888113528Z }, migration_in: None, migration_out: None }
20162024-10-22T00:37:20.932ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46992 req_id = 9592f13c-03b5-45db-aadf-a1d19c029430 response_code = 204 uri = /vpc-routes
20172024-10-22T00:37:21.099ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36780
20182024-10-22T00:37:21.100ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1022 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36780 req_id = 47e2bd6d-da5c-4b5d-92ba-c38e8fe5e026 response_code = 200 uri = /vmms/0aef4f71-019e-41bc-a78a-9c7292529cab/state
20192024-10-22T00:37:21.156ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51110
20202024-10-22T00:37:21.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51110 req_id = 18a7b9f4-8b08-46e7-857e-b7c773b53fba response_code = 200 uri = /vpc-routes
20212024-10-22T00:37:21.162ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60349
20222024-10-22T00:37:21.162ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60349 req_id = 317b082a-c332-46db-af03-87186a09647f response_code = 200 uri = /vmms/0aef4f71-019e-41bc-a78a-9c7292529cab/state
20232024-10-22T00:37:21.173ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41575
20242024-10-22T00:37:21.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41575 req_id = 74607911-c7df-4546-9648-d3cc391a05ed response_code = 200 uri = /vpc-routes
20252024-10-22T00:37:21.177ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {bb2562cc-387d-4dec-b6ab-7f8cb3916a5c: {10.151.2.175: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}, 10.151.2.176: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}}} }
20262024-10-22T00:37:21.177ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51110 req_id = 59aa7c71-cb57-4e27-a2a9-ff220e82e79a response_code = 204 uri = /eip-gateways
20272024-10-22T00:37:21.181ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39587
20282024-10-22T00:37:21.181ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39587 req_id = 8af52f24-c46c-4c9b-a1e6-e7a6c62bbf76 response_code = 200 uri = /v2p
20292024-10-22T00:37:21.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51110 req_id = 174af28d-f397-44c4-84d9-528866698d67 response_code = 204 uri = /vpc-routes
20302024-10-22T00:37:21.231ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {bb2562cc-387d-4dec-b6ab-7f8cb3916a5c: {10.151.2.175: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}, 10.151.2.176: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}}} }
20312024-10-22T00:37:21.231ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41575 req_id = 85f45f91-ed95-4c03-9730-321508ad0d75 response_code = 204 uri = /eip-gateways
20322024-10-22T00:37:21.249ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41575 req_id = f99845d9-2848-4b93-a519-56e516af887a response_code = 204 uri = /vpc-routes
20332024-10-22T00:37:21.270ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52561
20342024-10-22T00:37:21.270ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52561 req_id = 58d5cf62-ef54-4030-8914-ca7b72739868 response_code = 200 uri = /vmms/0aef4f71-019e-41bc-a78a-9c7292529cab/state
20352024-10-22T00:37:21.275ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61763
20362024-10-22T00:37:21.276ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
20372024-10-22T00:37:21.323ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
20382024-10-22T00:37:21.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110469 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61763 req_id = 5ce67ace-ce78-462a-80ee-082c85d6f921 response_code = 204 uri = /switch-ports
20392024-10-22T00:37:21.386ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37178
20402024-10-22T00:37:21.386ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62394
20412024-10-22T00:37:21.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 269 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61763 req_id = b51ec126-1f0c-4913-811f-cb235cb7894b response_code = 200 uri = /network-bootstore-config
20422024-10-22T00:37:21.387ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 211 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62394 req_id = bfb1a316-b05d-4d69-821a-4f3878eb544f response_code = 200 uri = /v2p
20432024-10-22T00:37:21.387ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
20442024-10-22T00:37:21.392ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39662
20452024-10-22T00:37:21.398ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39662 req_id = 4ce69713-2355-4405-b22d-8c232d626dd8 response_code = 200 uri = /vpc-routes
20462024-10-22T00:37:21.415ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {bb2562cc-387d-4dec-b6ab-7f8cb3916a5c: {10.151.2.175: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}, 10.151.2.176: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}}} }
20472024-10-22T00:37:21.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39662 req_id = c1ad2c58-deca-4618-98d5-a53541b39fe6 response_code = 204 uri = /eip-gateways
20482024-10-22T00:37:21.430ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
20492024-10-22T00:37:21.480ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46523
20502024-10-22T00:37:21.481ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 249 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46523 req_id = bf6d19e7-9ff6-48fe-b031-d1fa9ed72be8 response_code = 200 uri = /v2p
20512024-10-22T00:37:21.489ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 102726 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:37178 req_id = f2ec9ece-e6f6-4f8a-8e97-27ef8bec442b response_code = 204 uri = /switch-ports
20522024-10-22T00:37:21.490ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37178 req_id = 1801d57d-1ef6-4d72-8479-8f0eff8097ff response_code = 200 uri = /network-bootstore-config
20532024-10-22T00:37:21.490ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39662 req_id = acad418c-57cd-4f4a-b6bf-f34889aedd20 response_code = 204 uri = /vpc-routes
20542024-10-22T00:37:21.549ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61235
20552024-10-22T00:37:21.549ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4133
20562024-10-22T00:37:21.587ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4157
20572024-10-22T00:37:21.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93544 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61235 req_id = 9d16c069-4de2-401e-b83f-f6bc8d5564e6 response_code = 204 uri = /switch-ports
20582024-10-22T00:37:21.643ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61235 req_id = 9b3186db-cb52-4510-b421-b01ccf60e2de response_code = 200 uri = /network-bootstore-config
20592024-10-22T00:37:24.693ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9608 local_addr = [fd00:1122:3344:101::1]:54049 method = GET remote_addr = [fd00:1122:3344:101::d]:35322 req_id = e2883042-405e-4972-a7f2-80205802f067 response_code = 200 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a uri = /e50e0974-d68b-427b-915b-5bd9a655288a
20602024-10-22T00:37:27.904ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61830
20612024-10-22T00:37:27.904ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 29 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61830 req_id = 968273d9-1603-465f-b295-e4d218c1402f response_code = 200 uri = /baseboard
20622024-10-22T00:37:31.581ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40681
20632024-10-22T00:37:31.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 491 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40681 req_id = 7a23ecbe-ce6b-4ef0-ac07-31cc7a720883 response_code = 200 uri = /vmms/0aef4f71-019e-41bc-a78a-9c7292529cab/state
20642024-10-22T00:37:31.582ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-10-22T00:37:31.582066956Z } file = sled-agent/src/instance.rs:685 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20652024-10-22T00:37:31.582ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 7d023311-eb77-4625-832f-45e1831adeee new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-22T00:37:31.582066956Z } propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20662024-10-22T00:37:31.582ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-22T00:37:31.582066956Z }, migration_in: None, migration_out: None }
20672024-10-22T00:37:31.591ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-10-22T00:37:31.591429728Z } file = sled-agent/src/instance.rs:685 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20682024-10-22T00:37:31.591ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 7d023311-eb77-4625-832f-45e1831adeee new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-10-22T00:37:31.591429728Z } propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20692024-10-22T00:37:31.591ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:721 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20702024-10-22T00:37:31.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20712024-10-22T00:37:31.591ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 7510403784781763121 n_samples = 6 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
20722024-10-22T00:37:31.591ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 17994826670306291983 n_samples = 6 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
20732024-10-22T00:37:31.593ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20742024-10-22T00:37:31.593ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/042a0bdb-3057-4098-8648-7fd777304d2c/debug/bundle/zone", "/pool/int/20b2e8dd-3528-47c6-ab14-1bee850adf61/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/05815180-fa73-47c8-8793-9dad48a6a2a3/crypt/debug", "/pool/ext/4912870c-99c9-4864-9312-2d393933ce6a/crypt/debug", "/pool/ext/f643b80c-1fc2-4be9-93a9-fa7fd37e716e/crypt/debug", "/pool/ext/f3053f19-a930-43cc-8978-6dcb10f47d61/crypt/debug", "/pool/ext/d912c6f5-c454-43da-ae16-2d232f5f0eb5/crypt/debug", "/pool/ext/f72e2a71-8f15-4409-923c-af354b7632aa/crypt/debug", "/pool/ext/4124bd0f-6d92-452d-b8ec-890efde7c175/crypt/debug", "/pool/ext/6bea9bba-836b-44a8-977b-0b50b3178081/crypt/debug", "/pool/ext/c2c09b26-0ffd-409b-9a74-0a32dc4478f3/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab
20752024-10-22T00:37:32.431ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab", bundle_id: 56977bd8-8a99-4362-b563-dfc5707f7051 }, time_created: 2024-10-22T00:37:31.594102172Z, version: 0, cause: TerminatedInstance }
20762024-10-22T00:37:32.431ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_0aef4f71-019e-41bc-a78a-9c7292529cab file = sled-agent/src/instance.rs:935 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20772024-10-22T00:37:34.214ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20782024-10-22T00:37:34.230ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(15647631), kind: System }
20792024-10-22T00:37:34.230ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(15647631), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
20802024-10-22T00:37:34.230ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-22T00:37:34.230794309Z }, migration_in: None, migration_out: None }
20812024-10-22T00:37:34.307ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-22T00:37:34.230794309Z }, migration_in: None, migration_out: None }
20822024-10-22T00:37:34.312ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:872 instance_id = 7d023311-eb77-4625-832f-45e1831adeee propolis_id = 0aef4f71-019e-41bc-a78a-9c7292529cab
20832024-10-22T00:37:35.157ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64810
20842024-10-22T00:37:35.157ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64810 req_id = e6c0e132-65f6-47e3-807c-ebc269cb44f5 response_code = 200 uri = /vpc-routes
20852024-10-22T00:37:35.172ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {bb2562cc-387d-4dec-b6ab-7f8cb3916a5c: {10.151.2.176: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}, 10.151.2.175: {92faf8ff-4360-4bcc-bd21-22507ae7ee9d}}} }
20862024-10-22T00:37:35.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64810 req_id = 67460137-9cc8-4026-a021-fad99c9e306f response_code = 204 uri = /eip-gateways
20872024-10-22T00:37:35.180ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 180 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64810 req_id = c622832d-b91a-486f-8b97-e7cae93b673a response_code = 204 uri = /vpc-routes
20882024-10-22T00:37:35.205ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33836
20892024-10-22T00:37:35.205ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33836 req_id = 318b8e28-215f-4d87-ae0a-ff575813f7f4 response_code = 200 uri = /v2p
20902024-10-22T00:37:35.974ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49125
20912024-10-22T00:37:35.975ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49125 req_id = 7a978a58-9321-473a-993d-988df6e30eb7 response_code = 200 uri = /v2p
20922024-10-22T00:37:36.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59935
20932024-10-22T00:37:36.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59935 req_id = 5501688b-7227-4d82-ad06-8221f416f38b response_code = 200 uri = /vpc-routes
20942024-10-22T00:37:36.199ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
20952024-10-22T00:37:36.200ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1203 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a
20962024-10-22T00:37:36.200ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59935 req_id = aa0f4db6-24d5-40f6-a2d4-92315645f510 response_code = 204 uri = /eip-gateways
20972024-10-22T00:37:36.208ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 412 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59935 req_id = 83e1f641-0eec-47d8-9166-7ba7e59a85f5 response_code = 204 uri = /vpc-routes
20982024-10-22T00:37:37.567ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64025
20992024-10-22T00:37:37.567ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64025 req_id = ff32a519-18fe-430c-8305-e8ca837b34ee response_code = 200 uri = /vpc-routes
21002024-10-22T00:37:37.581ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1177 sled_id = e50e0974-d68b-427b-915b-5bd9a655288a values = ExternalIpGatewayMap { mappings: {} }
21012024-10-22T00:37:37.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64025 req_id = f2b5b0e2-ed61-4972-9888-715d797319b9 response_code = 204 uri = /eip-gateways
21022024-10-22T00:37:37.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 209 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64025 req_id = ee7e3e98-63da-41a7-a52a-404931ce0e35 response_code = 204 uri = /vpc-routes