Max level shown:
1 [ Sep 9 17:49:05 Enabled. ]
2 [ Sep 9 17:49:05 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Sep 9 17:49:05 Method "start" exited with status 0. ]
4 [ Sep 9 17:49:05 Rereading configuration. ]
5 [ Sep 9 17:49:05 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-09-09T17:49:05.869ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-09-09T17:49:05.869ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-09-09T17:49:05.921ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-09-09T17:49:05.921ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-09-09T17:49:06.041ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-09-09T17:49:06.051ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-09-09T17:49:06.059ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-09-09T17:49:06.663ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-09-09T17:49:06.801ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-09-09T17:49:06.802ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-09-09T17:49:06.802ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-09-09T17:49:06.802ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-09-09T17:49:06.802ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-09-09T17:49:06.802ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-09-09T17:49:06.805ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-09-09T17:49:06.805ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-09-09T17:49:06.805ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-09-09T17:49:06.805ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-09-09T17:49:06.805ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-09-09T17:49:06.805ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-09-09T17:49:06.805ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-09-09T17:49:06.805ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-09-09T17:49:06.805ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-09-09T17:49:06.816ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-09-09T17:49:06.816ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = d05e61ff-a727-4851-80ca-eab919525a00 (zpool)
322024-09-09T17:49:06.934ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal }
332024-09-09T17:49:07.414ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-09-09T17:49:07.414ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-09-09T17:49:07.414ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-09-09T17:49:07.414ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-09-09T17:49:07.414ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
382024-09-09T17:49:07.414ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
392024-09-09T17:49:07.414ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
402024-09-09T17:49:07.414ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
412024-09-09T17:49:07.414ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
422024-09-09T17:49:07.414ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
432024-09-09T17:49:07.423ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
442024-09-09T17:49:07.423ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool)
452024-09-09T17:49:07.533ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal }
462024-09-09T17:49:08.012ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
472024-09-09T17:49:08.012ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
482024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
492024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
502024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
512024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
522024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
532024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
542024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
552024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
562024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
572024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
582024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
592024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
602024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
612024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
622024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
632024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
642024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
652024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
662024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
672024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
682024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
692024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
702024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
712024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
722024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
732024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
742024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
752024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
762024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
772024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
782024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
792024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
802024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
812024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
822024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
832024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
862024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
882024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
892024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
902024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
912024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
922024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
942024-09-09T17:49:08.013ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
952024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
962024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
972024-09-09T17:49:08.013ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
982024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
992024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1002024-09-09T17:49:08.013ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
1012024-09-09T17:49:08.013ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
1022024-09-09T17:49:08.013ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1032024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1042024-09-09T17:49:08.013ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1052024-09-09T17:49:08.013ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1062024-09-09T17:49:08.013ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1072024-09-09T17:49:08.013ZINFOSledAgent (bootstore): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-09-09T17:49:08.013ZINFOSledAgent (bootstore): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-09-09T17:49:08.013ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1102024-09-09T17:49:08.013ZINFOSledAgent (bootstore): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1112024-09-09T17:49:08.013ZINFOSledAgent (bootstore): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1122024-09-09T17:49:08.013ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1132024-09-09T17:49:08.013ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1142024-09-09T17:49:08.013ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1152024-09-09T17:49:08.024ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:730
1162024-09-09T17:49:08.024ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 34069, tv_nsec: 79059468 } time_to_next_cleanup = 599.99993284s
1172024-09-09T17:49:08.024ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3866
1182024-09-09T17:49:08.024ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1192024-09-09T17:49:08.024ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/sled-agent-request.json file = common/src/ledger.rs:188
1202024-09-09T17:49:08.024ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/sled-agent-request.json file = common/src/ledger.rs:188
1212024-09-09T17:49:08.027ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1222024-09-09T17:49:08.027ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1232024-09-09T17:49:08.027ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1053
1242024-09-09T17:49:08.027ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1252024-09-09T17:49:08.027ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1262024-09-09T17:49:08.027ZINFOSledAgent (RSS): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1272024-09-09T17:49:08.027ZINFOSledAgent (RSS): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1282024-09-09T17:49:08.027ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1113
1292024-09-09T17:49:08.027ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1302024-09-09T17:49:08.027ZINFOSledAgent (RSS): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/rss-sled-plan.json file = common/src/ledger.rs:188
1312024-09-09T17:49:08.027ZINFOSledAgent (RSS): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/rss-sled-plan.json file = common/src/ledger.rs:188
1322024-09-09T17:49:08.027ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1154
1332024-09-09T17:49:08.027ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1342024-09-09T17:49:08.027ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1352024-09-09T17:49:08.027ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1362024-09-09T17:49:08.027ZINFOSledAgent (RSS): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1372024-09-09T17:49:08.028ZINFOSledAgent (RSS): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1382024-09-09T17:49:08.028ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 1068bc7e-e3ab-44f6-b097-299854b0081f, sleds: { [fdb0:18c0:4d0d:9fb2::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 61e07fcc-45b4-4b25-ae0d-e78eee62eb60, rack_id: 1068bc7e-e3ab-44f6-b097-299854b0081f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1392024-09-09T17:49:08.028ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1187
1402024-09-09T17:49:08.028ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1412024-09-09T17:49:08.028ZINFOSledAgent (bootstore): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1422024-09-09T17:49:08.028ZINFOSledAgent (bootstore): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1432024-09-09T17:49:08.028ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1442024-09-09T17:49:08.028ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 61e07fcc-45b4-4b25-ae0d-e78eee62eb60, rack_id: 1068bc7e-e3ab-44f6-b097-299854b0081f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1452024-09-09T17:49:08.029ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:42816
1462024-09-09T17:49:08.029ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 61e07fcc-45b4-4b25-ae0d-e78eee62eb60, rack_id: 1068bc7e-e3ab-44f6-b097-299854b0081f, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1472024-09-09T17:49:08.029ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1482024-09-09T17:49:08.029ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1492024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1502024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1512024-09-09T17:49:08.029ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1522024-09-09T17:49:08.029ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1532024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1542024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1552024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1562024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1572024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1582024-09-09T17:49:08.029ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1592024-09-09T17:49:08.029ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1602024-09-09T17:49:08.031ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4163
1612024-09-09T17:49:08.031ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4545
1622024-09-09T17:49:08.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1632024-09-09T17:49:08.082ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1642024-09-09T17:49:08.098ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1652024-09-09T17:49:08.107ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:388 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1662024-09-09T17:49:08.107ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1672024-09-09T17:49:08.107ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:400 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1682024-09-09T17:49:08.107ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1692024-09-09T17:49:08.111ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 74.392977ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9587c68f-9ed8-4514-8ce1-e43a7c989fd2", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9587c68f-9ed8-4514-8ce1-e43a7c989fd2" } file = clients/ddm-admin-client/src/lib.rs:177
1702024-09-09T17:49:08.121ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_d05e61ff-a727-4851-80ca-eab919525a00/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1712024-09-09T17:49:08.147ZINFOSledAgent: successfully created encrypted zvol: name="oxi_d05e61ff-a727-4851-80ca-eab919525a00/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1722024-09-09T17:49:08.147ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_d05e61ff-a727-4851-80ca-eab919525a00/swap" file = sled-agent/src/swap_device.rs:96
1732024-09-09T17:49:08.152ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:415 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1742024-09-09T17:49:08.152ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:127
1752024-09-09T17:49:08.152ZINFOSledAgent (BackingFs): Ensuring dataset oxi_d05e61ff-a727-4851-80ca-eab919525a00/backing/fmd file = sled-agent/src/backing_fs.rs:137
1762024-09-09T17:49:08.186ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-09-09T17:49:08.187ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 102.794585ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9b7f7d51-caee-4183-8d91-71af36654c86", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9b7f7d51-caee-4183-8d91-71af36654c86" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-09-09T17:49:08.258ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:172
1792024-09-09T17:49:08.291ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1802024-09-09T17:49:08.292ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 175.285683ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8e2b7901-dfd2-4a90-9e4a-c2e9a451d984", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8e2b7901-dfd2-4a90-9e4a-c2e9a451d984" } file = clients/ddm-admin-client/src/lib.rs:177
1812024-09-09T17:49:08.377ZINFOSledAgent (BackingFs): Mounting oxi_d05e61ff-a727-4851-80ca-eab919525a00/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:180
1822024-09-09T17:49:08.394ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:188
1832024-09-09T17:49:08.394ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:188
1842024-09-09T17:49:08.394ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:188
1852024-09-09T17:49:08.394ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:194
1862024-09-09T17:49:08.468ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1872024-09-09T17:49:08.468ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 183.003904ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "34f93ef3-66a1-422e-bd08-bf1e93412bdd", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "34f93ef3-66a1-422e-bd08-bf1e93412bdd" } file = clients/ddm-admin-client/src/lib.rs:177
1882024-09-09T17:49:08.652ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892024-09-09T17:49:08.653ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 268.840739ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6f0eaebb-6551-4f3b-abcb-f26409ad2ca7", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6f0eaebb-6551-4f3b-abcb-f26409ad2ca7" } file = clients/ddm-admin-client/src/lib.rs:177
1902024-09-09T17:49:08.923ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-09-09T17:49:08.924ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 195.657973ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ac54d7b9-1e16-4725-871b-68aa9abd35e0", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ac54d7b9-1e16-4725-871b-68aa9abd35e0" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-09-09T17:49:09.114ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1932024-09-09T17:49:09.120ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:42327 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1942024-09-09T17:49:09.120ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1952024-09-09T17:49:09.120ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:42327 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1962024-09-09T17:49:09.121ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1972024-09-09T17:49:09.121ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 203.032926ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1982024-09-09T17:49:09.122ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
1992024-09-09T17:49:09.122ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2002024-09-09T17:49:09.122ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:914 underlay_address = fd00:1122:3344:101::1
2012024-09-09T17:49:09.122ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
2022024-09-09T17:49:09.326ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 423.731904ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2032024-09-09T17:49:09.751ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.079422325s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2042024-09-09T17:49:10.832ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.844026654s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2052024-09-09T17:49:11.681ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2639
2062024-09-09T17:49:11.681ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2857
2072024-09-09T17:49:11.681ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2509
2082024-09-09T17:49:11.681ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2556
2092024-09-09T17:49:11.681ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2938
2102024-09-09T17:49:11.681ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2980
2112024-09-09T17:49:11.683ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2782
2122024-09-09T17:49:11.684ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2550
2132024-09-09T17:49:11.685ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='fdca555f-3b9a-4118-ba07-7428edbb0dfc'/> <propval type="astring" name="rack_id" value='1068bc7e-e3ab-44f6-b097-299854b0081f'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fecf:45d6'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='61e07fcc-45b4-4b25-ae0d-e78eee62eb60'/> <propval type="astring" name="rack_id" value='1068bc7e-e3ab-44f6-b097-299854b0081f'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='61e07fcc-45b4-4b25-ae0d-e78eee62eb60'/> <propval type="astring" name="rack_uuid" value='1068bc7e-e3ab-44f6-b097-299854b0081f'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='61e07fcc-45b4-4b25-ae0d-e78eee62eb60'/> <propval type="astring" name="rack_uuid" value='1068bc7e-e3ab-44f6-b097-299854b0081f'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2142024-09-09T17:49:11.685ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2152024-09-09T17:49:11.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 61.026599ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2162024-09-09T17:49:12.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 74.933565ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2172024-09-09T17:49:12.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 149.408513ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2182024-09-09T17:49:12.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 499.404536ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-09-09T17:49:13.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 690.923478ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-09-09T17:49:13.678ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 5.16935289s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2212024-09-09T17:49:13.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 543.532017ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-09-09T17:49:14.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 975.535938ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-09-09T17:49:15.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 822.898411ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-09-09T17:49:16.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 766.861436ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-09-09T17:49:17.576ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2262024-09-09T17:49:17.577ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2272024-09-09T17:49:17.577ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2282024-09-09T17:49:17.577ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2292024-09-09T17:49:17.577ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2302024-09-09T17:49:17.577ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2312024-09-09T17:49:17.577ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:42816
2322024-09-09T17:49:17.577ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2332024-09-09T17:49:17.577ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3866
2342024-09-09T17:49:17.577ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2352024-09-09T17:49:17.577ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2362024-09-09T17:49:17.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2372024-09-09T17:49:17.863ZINFOSledAgent (RSS): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2382024-09-09T17:49:17.863ZINFOSledAgent (RSS): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2392024-09-09T17:49:17.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2402024-09-09T17:49:17.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-09-09T17:49:17.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2422024-09-09T17:49:17.868ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4187 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2432024-09-09T17:49:17.868ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4203
2442024-09-09T17:49:17.868ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2452024-09-09T17:49:18.164ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60350
2462024-09-09T17:49:18.164ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2472024-09-09T17:49:18.164ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60350 req_id = 7bd72791-e2af-41c6-a06c-a4d1e99bee76 response_code = 200 uri = /inventory
2482024-09-09T17:49:18.195ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39666
2492024-09-09T17:49:18.195ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 36 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39666 req_id = bff3fc1f-8269-4508-b125-a761a8a9b98a response_code = 200 uri = /sled-role
2502024-09-09T17:49:18.195ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2512024-09-09T17:49:18.195ZINFOSledAgent (RSS): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2522024-09-09T17:49:18.196ZINFOSledAgent (RSS): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2532024-09-09T17:49:18.196ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:1001
2542024-09-09T17:49:18.236ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 99445748-3f8a-47eb-b03d-540bb23b7924, pool_id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: eaf34cef-e592-4e7c-b9f3-1a240bd6bcf6, pool_id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: b3e70874-57ef-4fcb-9f44-a18b6a655d68, pool_id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 3cca5293-4433-4956-b75c-0e6967d69a04, pool_id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 032350b3-5ef6-49e0-82b7-c2f6c5c3d580, pool_id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 8d6a3fc4-42ae-4137-8a4c-0c429a53310f, pool_id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 31811878-08d9-4dd7-82d9-61bcc051e746, pool_id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 6c038f91-337d-4974-956d-da4fd59787a8, pool_id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: e6df29c4-987d-4c4b-8971-6886a64153eb, pool_id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:355 sled_address = [fd00:1122:3344:101::1]:12345
2552024-09-09T17:49:18.236ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33471
2562024-09-09T17:49:18.236ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:856 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2572024-09-09T17:49:18.236ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 99445748-3f8a-47eb-b03d-540bb23b7924, pool_id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: eaf34cef-e592-4e7c-b9f3-1a240bd6bcf6, pool_id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: b3e70874-57ef-4fcb-9f44-a18b6a655d68, pool_id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 3cca5293-4433-4956-b75c-0e6967d69a04, pool_id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 032350b3-5ef6-49e0-82b7-c2f6c5c3d580, pool_id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 8d6a3fc4-42ae-4137-8a4c-0c429a53310f, pool_id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 31811878-08d9-4dd7-82d9-61bcc051e746, pool_id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 6c038f91-337d-4974-956d-da4fd59787a8, pool_id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: e6df29c4-987d-4c4b-8971-6886a64153eb, pool_id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2582024-09-09T17:49:18.236ZINFOSledAgent (StorageManager): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2592024-09-09T17:49:18.236ZINFOSledAgent (StorageManager): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2602024-09-09T17:49:18.236ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2612024-09-09T17:49:18.236ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2622024-09-09T17:49:18.236ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2632024-09-09T17:49:18.236ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2642024-09-09T17:49:18.236ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2652024-09-09T17:49:18.384ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2662024-09-09T17:49:18.384ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool)
2672024-09-09T17:49:18.656ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/61e07fcc-45b4-4b25-ae0d-e78eee62eb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2682024-09-09T17:49:18.895ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 6.627662798s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2692024-09-09T17:49:19.598ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2702024-09-09T17:49:19.617ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }
2712024-09-09T17:49:19.625ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:218
2722024-09-09T17:49:19.625ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:220
2732024-09-09T17:49:19.625ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:224
2742024-09-09T17:49:19.625ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:226
2752024-09-09T17:49:19.625ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2762024-09-09T17:49:19.626ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2772024-09-09T17:49:19.626ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
2782024-09-09T17:49:19.659ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/61e07fcc-45b4-4b25-ae0d-e78eee62eb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
2792024-09-09T17:49:19.668ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2802024-09-09T17:49:19.676ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e68e093e-0654-4278-a79f-9c234ab4cc35/crypt/zone file = sled-storage/src/dataset.rs:281
2812024-09-09T17:49:19.886ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e68e093e-0654-4278-a79f-9c234ab4cc35 file = sled-storage/src/dataset.rs:367
2822024-09-09T17:49:19.894ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2832024-09-09T17:49:19.894ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2842024-09-09T17:49:19.894ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2852024-09-09T17:49:19.894ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2862024-09-09T17:49:19.894ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2872024-09-09T17:49:19.904ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2882024-09-09T17:49:19.904ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool)
2892024-09-09T17:49:20.024ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }
2902024-09-09T17:49:20.033ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:218
2912024-09-09T17:49:20.033ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:220
2922024-09-09T17:49:20.033ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:224
2932024-09-09T17:49:20.033ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:226
2942024-09-09T17:49:20.033ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2952024-09-09T17:49:20.033ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2962024-09-09T17:49:20.033ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
2972024-09-09T17:49:20.074ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2982024-09-09T17:49:20.082ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f5eda459-d1e9-44f5-b731-f6490e3fc719/crypt/zone file = sled-storage/src/dataset.rs:281
2992024-09-09T17:49:20.283ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f5eda459-d1e9-44f5-b731-f6490e3fc719 file = sled-storage/src/dataset.rs:367
3002024-09-09T17:49:20.291ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3012024-09-09T17:49:20.291ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3022024-09-09T17:49:20.291ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3032024-09-09T17:49:20.291ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3042024-09-09T17:49:20.291ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3052024-09-09T17:49:20.301ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3062024-09-09T17:49:20.301ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool)
3072024-09-09T17:49:20.421ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }
3082024-09-09T17:49:20.429ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:218
3092024-09-09T17:49:20.429ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:220
3102024-09-09T17:49:20.429ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:224
3112024-09-09T17:49:20.429ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:226
3122024-09-09T17:49:20.429ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3132024-09-09T17:49:20.429ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3142024-09-09T17:49:20.429ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3152024-09-09T17:49:20.466ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3162024-09-09T17:49:20.474ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_677df3fa-a1f5-4e63-93ee-592a272d5e3f/crypt/zone file = sled-storage/src/dataset.rs:281
3172024-09-09T17:49:20.665ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/61e07fcc-45b4-4b25-ae0d-e78eee62eb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
3182024-09-09T17:49:20.675ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_677df3fa-a1f5-4e63-93ee-592a272d5e3f file = sled-storage/src/dataset.rs:367
3192024-09-09T17:49:20.683ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3202024-09-09T17:49:20.683ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3212024-09-09T17:49:20.683ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3222024-09-09T17:49:20.683ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3232024-09-09T17:49:20.683ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3242024-09-09T17:49:20.692ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3252024-09-09T17:49:20.692ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool)
3262024-09-09T17:49:20.812ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }
3272024-09-09T17:49:20.820ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:218
3282024-09-09T17:49:20.820ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:220
3292024-09-09T17:49:20.820ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:224
3302024-09-09T17:49:20.820ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:226
3312024-09-09T17:49:20.820ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3322024-09-09T17:49:20.820ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3332024-09-09T17:49:20.820ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3342024-09-09T17:49:20.862ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3352024-09-09T17:49:20.870ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crypt/zone file = sled-storage/src/dataset.rs:281
3362024-09-09T17:49:21.076ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582 file = sled-storage/src/dataset.rs:367
3372024-09-09T17:49:21.084ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3382024-09-09T17:49:21.084ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3392024-09-09T17:49:21.084ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3402024-09-09T17:49:21.084ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3412024-09-09T17:49:21.084ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3422024-09-09T17:49:21.094ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3432024-09-09T17:49:21.094ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool)
3442024-09-09T17:49:21.214ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }
3452024-09-09T17:49:21.221ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:218
3462024-09-09T17:49:21.221ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:220
3472024-09-09T17:49:21.221ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:224
3482024-09-09T17:49:21.221ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:226
3492024-09-09T17:49:21.221ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3502024-09-09T17:49:21.221ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3512024-09-09T17:49:21.221ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3522024-09-09T17:49:21.261ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3532024-09-09T17:49:21.270ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_46b15cce-a870-48ea-aab8-5f90f01b99e4/crypt/zone file = sled-storage/src/dataset.rs:281
3542024-09-09T17:49:21.481ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_46b15cce-a870-48ea-aab8-5f90f01b99e4 file = sled-storage/src/dataset.rs:367
3552024-09-09T17:49:21.489ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3562024-09-09T17:49:21.489ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3572024-09-09T17:49:21.489ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3582024-09-09T17:49:21.489ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3592024-09-09T17:49:21.489ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3602024-09-09T17:49:21.499ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3612024-09-09T17:49:21.499ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool)
3622024-09-09T17:49:21.615ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool), kind: External }
3632024-09-09T17:49:21.623ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:218
3642024-09-09T17:49:21.623ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:220
3652024-09-09T17:49:21.623ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:224
3662024-09-09T17:49:21.623ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:226
3672024-09-09T17:49:21.623ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3682024-09-09T17:49:21.623ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3692024-09-09T17:49:21.623ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3702024-09-09T17:49:21.664ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3712024-09-09T17:49:21.668ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/61e07fcc-45b4-4b25-ae0d-e78eee62eb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
3722024-09-09T17:49:21.673ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_0720cae8-ac21-4fb2-9b27-8f836e4c5c41/crypt/zone file = sled-storage/src/dataset.rs:281
3732024-09-09T17:49:21.899ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_0720cae8-ac21-4fb2-9b27-8f836e4c5c41 file = sled-storage/src/dataset.rs:367
3742024-09-09T17:49:21.907ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3752024-09-09T17:49:21.907ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3762024-09-09T17:49:21.907ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3772024-09-09T17:49:21.907ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3782024-09-09T17:49:21.907ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3792024-09-09T17:49:21.918ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3802024-09-09T17:49:21.918ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool)
3812024-09-09T17:49:22.042ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool), kind: External }
3822024-09-09T17:49:22.050ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:218
3832024-09-09T17:49:22.050ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:220
3842024-09-09T17:49:22.050ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:224
3852024-09-09T17:49:22.050ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:226
3862024-09-09T17:49:22.050ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3872024-09-09T17:49:22.051ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3882024-09-09T17:49:22.051ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3892024-09-09T17:49:22.090ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3902024-09-09T17:49:22.096ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299/crypt/zone file = sled-storage/src/dataset.rs:281
3912024-09-09T17:49:22.292ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 file = sled-storage/src/dataset.rs:367
3922024-09-09T17:49:22.299ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3932024-09-09T17:49:22.299ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3942024-09-09T17:49:22.299ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3952024-09-09T17:49:22.299ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3962024-09-09T17:49:22.299ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3972024-09-09T17:49:22.309ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3982024-09-09T17:49:22.309ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool)
3992024-09-09T17:49:22.433ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool), kind: External }
4002024-09-09T17:49:22.442ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:218
4012024-09-09T17:49:22.442ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:220
4022024-09-09T17:49:22.442ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:224
4032024-09-09T17:49:22.442ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:226
4042024-09-09T17:49:22.442ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4052024-09-09T17:49:22.442ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4062024-09-09T17:49:22.442ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
4072024-09-09T17:49:22.482ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4082024-09-09T17:49:22.490ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2c079939-1543-4f7b-ab38-d421cbf95e07/crypt/zone file = sled-storage/src/dataset.rs:281
4092024-09-09T17:49:22.671ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/61e07fcc-45b4-4b25-ae0d-e78eee62eb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4102024-09-09T17:49:22.698ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2c079939-1543-4f7b-ab38-d421cbf95e07 file = sled-storage/src/dataset.rs:367
4112024-09-09T17:49:22.706ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4122024-09-09T17:49:22.706ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4132024-09-09T17:49:22.706ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4142024-09-09T17:49:22.706ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4152024-09-09T17:49:22.707ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4162024-09-09T17:49:22.717ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4172024-09-09T17:49:22.717ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool)
4182024-09-09T17:49:22.834ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool), kind: External }
4192024-09-09T17:49:22.842ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:218
4202024-09-09T17:49:22.842ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:220
4212024-09-09T17:49:22.842ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:224
4222024-09-09T17:49:22.842ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:226
4232024-09-09T17:49:22.842ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4242024-09-09T17:49:22.842ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4252024-09-09T17:49:22.842ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
4262024-09-09T17:49:22.882ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4272024-09-09T17:49:22.890ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_b6fbc8d5-08ec-4cd5-ae5d-395890772649/crypt/zone file = sled-storage/src/dataset.rs:281
4282024-09-09T17:49:23.097ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_b6fbc8d5-08ec-4cd5-ae5d-395890772649 file = sled-storage/src/dataset.rs:367
4292024-09-09T17:49:23.105ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4302024-09-09T17:49:23.105ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4312024-09-09T17:49:23.105ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4322024-09-09T17:49:23.105ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d05e61ff-a727-4851-80ca-eab919525a00 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6830ec9d-22cb-45c0-ba9d-5a3728bef44f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4332024-09-09T17:49:23.105ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4342024-09-09T17:49:23.105ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4352024-09-09T17:49:23.105ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4362024-09-09T17:49:23.105ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:860 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4372024-09-09T17:49:23.105ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4382024-09-09T17:49:23.106ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:886 generation = Generation(1) sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4392024-09-09T17:49:23.106ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:891 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4402024-09-09T17:49:23.106ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:896 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4412024-09-09T17:49:23.106ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4422024-09-09T17:49:23.106ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:902 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4432024-09-09T17:49:23.106ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:803 gen = Generation(1)
4442024-09-09T17:49:23.106ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:907 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4452024-09-09T17:49:23.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4869600 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33471 req_id = 524bd81d-250f-456c-b856-d78a2e7790ac response_code = 200 uri = /omicron-physical-disks
4462024-09-09T17:49:23.138ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
4472024-09-09T17:49:23.138ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57103
4482024-09-09T17:49:23.138ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4492024-09-09T17:49:23.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4502024-09-09T17:49:23.198ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4512024-09-09T17:49:23.198ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4522024-09-09T17:49:23.258ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4532024-09-09T17:49:23.258ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
4542024-09-09T17:49:23.317ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4552024-09-09T17:49:23.317ZINFOSledAgent (ServiceManager): No ledger in /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:188
4562024-09-09T17:49:23.317ZINFOSledAgent (ServiceManager): No ledger in /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:188
4572024-09-09T17:49:23.317ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4582024-09-09T17:49:23.361ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4592024-09-09T17:49:23.361ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4602024-09-09T17:49:23.361ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4612024-09-09T17:49:23.415ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703 file = illumos-utils/src/zone.rs:316
4622024-09-09T17:49:23.419ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003 file = illumos-utils/src/zone.rs:316
4632024-09-09T17:49:23.423ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4 file = illumos-utils/src/zone.rs:316
4642024-09-09T17:49:23.435ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703 file = illumos-utils/src/zone.rs:354
4652024-09-09T17:49:23.441ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003 file = illumos-utils/src/zone.rs:354
4662024-09-09T17:49:23.445ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4 file = illumos-utils/src/zone.rs:354
4672024-09-09T17:49:23.674ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/61e07fcc-45b4-4b25-ae0d-e78eee62eb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
4682024-09-09T17:49:24.943ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4692024-09-09T17:49:24.943ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4702024-09-09T17:49:25.422ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4712024-09-09T17:49:25.422ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
4722024-09-09T17:49:25.909ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4732024-09-09T17:49:25.909ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4742024-09-09T17:49:25.909ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4752024-09-09T17:49:25.909ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
4762024-09-09T17:49:25.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 63.545761ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
4772024-09-09T17:49:25.922ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
4782024-09-09T17:49:25.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 111.603877ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
4792024-09-09T17:49:26.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 34.57806ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
4802024-09-09T17:49:26.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 47.969746ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
4812024-09-09T17:49:26.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 145.358228ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
4822024-09-09T17:49:26.113ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 137.204582ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
4832024-09-09T17:49:26.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 217.346502ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
4842024-09-09T17:49:26.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 240.786735ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
4852024-09-09T17:49:26.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 257.547097ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
4862024-09-09T17:49:26.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 366.123874ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
4872024-09-09T17:49:26.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 383.559119ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
4882024-09-09T17:49:26.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 302.465992ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
4892024-09-09T17:49:26.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 909.190125ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
4902024-09-09T17:49:26.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 565.764198ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
4912024-09-09T17:49:26.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 1.153512659s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
4922024-09-09T17:49:27.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 1.392253659s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
4932024-09-09T17:49:27.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 513.263998ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
4942024-09-09T17:49:27.936ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4211
4952024-09-09T17:49:27.936ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4221 gateway = fd00:1122:3344:101::1
4962024-09-09T17:49:27.943ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4300
4972024-09-09T17:49:28.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 872.838512ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
4982024-09-09T17:49:28.091ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4348
4992024-09-09T17:49:28.091ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4375
5002024-09-09T17:49:28.135ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4388
5012024-09-09T17:49:28.135ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4252
5022024-09-09T17:49:28.198ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4294
5032024-09-09T17:49:28.198ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4355 rack_subnet = fd00:1122:3344::
5042024-09-09T17:49:28.219ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4366
5052024-09-09T17:49:28.219ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4408
5062024-09-09T17:49:28.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 1.038561977s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
5072024-09-09T17:49:28.320ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4440
5082024-09-09T17:49:28.320ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4446
5092024-09-09T17:49:28.437ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:62285
5102024-09-09T17:49:28.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:62285 req_id = 085d8efc-b258-4cae-ae34-45b7e8d875ce response_code = 200 uri = /sled-identifiers
5112024-09-09T17:49:28.454ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4487
5122024-09-09T17:49:28.454ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5132024-09-09T17:49:28.487ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5142024-09-09T17:49:28.520ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5152024-09-09T17:49:28.520ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5162024-09-09T17:49:28.520ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5172024-09-09T17:49:28.520ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5182024-09-09T17:49:28.556ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
5192024-09-09T17:49:28.605ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5202024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5212024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5222024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5232024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5242024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5252024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5262024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5272024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5282024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5292024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5302024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5312024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5322024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5332024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5342024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5352024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5362024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5372024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5382024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5392024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5402024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5412024-09-09T17:49:28.605ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5422024-09-09T17:49:28.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 615.59652ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
5432024-09-09T17:49:28.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 1.214967533s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
5442024-09-09T17:49:29.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 628.303053ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
5452024-09-09T17:49:29.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 1.332949303s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
5462024-09-09T17:49:29.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003") failed: Property not found. retry in 991.329076ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
5472024-09-09T17:49:30.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 729.867972ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
5482024-09-09T17:49:30.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4") failed: Property not found. retry in 1.028685102s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
5492024-09-09T17:49:30.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703") failed: Property not found. retry in 758.219575ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
5502024-09-09T17:49:31.871ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_internal_dns_21f5073c-362d-497d-ae9e-f6ab85374703
5512024-09-09T17:49:31.871ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_internal_dns_42b0964a-b1a3-443e-b006-c18e1ad8d003
5522024-09-09T17:49:31.871ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_internal_dns_9704e6fa-a599-4b20-92b9-bc19ae282ee4
5532024-09-09T17:49:31.871ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5542024-09-09T17:49:31.871ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5552024-09-09T17:49:31.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8733506 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:57103 req_id = 4887cabd-9cf2-4063-876c-c089d45ae797 response_code = 204 uri = /omicron-zones
5562024-09-09T17:49:31.873ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5572024-09-09T17:49:31.930ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5582024-09-09T17:49:31.985ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5592024-09-09T17:49:32.038ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:591
5602024-09-09T17:49:32.038ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5612024-09-09T17:49:32.071ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5622024-09-09T17:49:32.071ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5632024-09-09T17:49:32.071ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5642024-09-09T17:49:32.101ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
5652024-09-09T17:49:32.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52480
5662024-09-09T17:49:32.101ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5672024-09-09T17:49:32.101ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5682024-09-09T17:49:32.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5692024-09-09T17:49:32.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5702024-09-09T17:49:32.138ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5712024-09-09T17:49:32.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
5722024-09-09T17:49:32.154ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5732024-09-09T17:49:32.155ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
5742024-09-09T17:49:32.155ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
5752024-09-09T17:49:32.155ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5762024-09-09T17:49:32.166ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5772024-09-09T17:49:32.198ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5782024-09-09T17:49:32.199ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5792024-09-09T17:49:32.199ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5802024-09-09T17:49:32.229ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 220, 173]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5812024-09-09T17:49:32.229ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = boundary_ntp
5822024-09-09T17:49:32.230ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5832024-09-09T17:49:32.230ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5842024-09-09T17:49:32.230ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5852024-09-09T17:49:32.230ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5862024-09-09T17:49:32.252ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea file = illumos-utils/src/zone.rs:316
5872024-09-09T17:49:32.270ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea file = illumos-utils/src/zone.rs:354
5882024-09-09T17:49:32.903ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5892024-09-09T17:49:32.903ZINFOSledAgent (ServiceManager): Booting oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5902024-09-09T17:49:33.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 60.81027ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5912024-09-09T17:49:33.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 134.412399ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5922024-09-09T17:49:33.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 226.876859ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5932024-09-09T17:49:33.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 498.367216ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5942024-09-09T17:49:34.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 1.101270167s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5952024-09-09T17:49:35.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 944.547908ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5962024-09-09T17:49:36.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 966.565671ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5972024-09-09T17:49:37.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea") failed: Property not found. retry in 942.041728ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5982024-09-09T17:49:38.027ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea
5992024-09-09T17:49:38.027ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6002024-09-09T17:49:38.028ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6012024-09-09T17:49:38.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5927006 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52480 req_id = c7f3e36b-57c0-4da6-8e5a-f5f0b8358d5e response_code = 204 uri = /omicron-zones
6022024-09-09T17:49:38.030ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:632
6032024-09-09T17:49:38.060ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6042024-09-09T17:49:38.060ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48727
6052024-09-09T17:49:38.070ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6062024-09-09T17:49:38.070ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 9927 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48727 req_id = ceda5413-e0c0-4e31-a3c5-d6326b134a11 response_code = 500 uri = /timesync
6072024-09-09T17:49:38.072ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6082024-09-09T17:49:38.230ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6092024-09-09T17:49:38.231ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54306
6102024-09-09T17:49:42.240ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6112024-09-09T17:49:42.240ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4009016 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54306 req_id = 2f61ba36-28b8-4528-9203-757c67f8640e response_code = 500 uri = /timesync
6122024-09-09T17:49:42.243ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6132024-09-09T17:49:42.422ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6142024-09-09T17:49:42.422ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38334
6152024-09-09T17:49:42.431ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6162024-09-09T17:49:42.432ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8960 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38334 req_id = 6c20c0a6-01da-419a-a80e-e0f77d453e44 response_code = 500 uri = /timesync
6172024-09-09T17:49:42.433ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6182024-09-09T17:49:42.637ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6192024-09-09T17:49:42.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42238
6202024-09-09T17:49:42.646ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_30c18989-6f47-40e1-ab15-11c78531e2ea': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6212024-09-09T17:49:42.646ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8875 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42238 req_id = fb3b1f3f-e9e6-448d-9463-f7d6f78875dd response_code = 500 uri = /timesync
6222024-09-09T17:49:42.648ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6232024-09-09T17:49:42.873ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6242024-09-09T17:49:42.873ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33798
6252024-09-09T17:49:46.163ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4021751 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33798 req_id = ef36abe1-aafd-40f4-aaa1-c00efe7c538d response_code = 200 uri = /timesync
6262024-09-09T17:49:46.165ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 3494071282, ip_addr: 208.67.75.242, stratum: 3, ref_time: 1725904185.9544878, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6272024-09-09T17:49:46.165ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1266
6282024-09-09T17:49:46.196ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6292024-09-09T17:49:46.196ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36776
6302024-09-09T17:49:46.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6312024-09-09T17:49:46.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6322024-09-09T17:49:46.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6332024-09-09T17:49:46.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6342024-09-09T17:49:46.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6352024-09-09T17:49:46.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
6362024-09-09T17:49:46.252ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6372024-09-09T17:49:46.252ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6382024-09-09T17:49:46.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6392024-09-09T17:49:46.316ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6402024-09-09T17:49:46.381ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6412024-09-09T17:49:46.381ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
6422024-09-09T17:49:46.443ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6432024-09-09T17:49:46.443ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6442024-09-09T17:49:46.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6452024-09-09T17:49:46.504ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6462024-09-09T17:49:46.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6472024-09-09T17:49:46.574ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
6482024-09-09T17:49:46.574ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
6492024-09-09T17:49:46.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6502024-09-09T17:49:46.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
6512024-09-09T17:49:46.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
6522024-09-09T17:49:46.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
6532024-09-09T17:49:46.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
6542024-09-09T17:49:46.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
6552024-09-09T17:49:46.606ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 410531 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 63ab3736-faff-4ad6-b493-7190e218a1d5 response_code = 500 uri = /omicron-zones
6562024-09-09T17:49:46.607ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "63ab3736-faff-4ad6-b493-7190e218a1d5", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "63ab3736-faff-4ad6-b493-7190e218a1d5" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 136.95336ms sled_address = [fd00:1122:3344:101::1]:12345
6572024-09-09T17:49:46.745ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6582024-09-09T17:49:46.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6592024-09-09T17:49:46.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6602024-09-09T17:49:46.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6612024-09-09T17:49:46.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6622024-09-09T17:49:46.781ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6632024-09-09T17:49:46.781ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
6642024-09-09T17:49:46.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6652024-09-09T17:49:46.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6662024-09-09T17:49:46.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6672024-09-09T17:49:46.816ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6682024-09-09T17:49:46.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6692024-09-09T17:49:46.834ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
6702024-09-09T17:49:46.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6712024-09-09T17:49:46.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6722024-09-09T17:49:46.868ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6732024-09-09T17:49:46.868ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6742024-09-09T17:49:46.887ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6752024-09-09T17:49:46.887ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
6762024-09-09T17:49:46.887ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
6772024-09-09T17:49:46.887ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6782024-09-09T17:49:46.918ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
6792024-09-09T17:49:46.918ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
6802024-09-09T17:49:46.918ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
6812024-09-09T17:49:46.919ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
6822024-09-09T17:49:46.919ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
6832024-09-09T17:49:46.919ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 173533 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = d9b93661-f54c-48a5-9a74-1b4ee14dd11b response_code = 500 uri = /omicron-zones
6842024-09-09T17:49:46.919ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d9b93661-f54c-48a5-9a74-1b4ee14dd11b", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d9b93661-f54c-48a5-9a74-1b4ee14dd11b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 64.626552ms sled_address = [fd00:1122:3344:101::1]:12345
6852024-09-09T17:49:46.984ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6862024-09-09T17:49:46.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6872024-09-09T17:49:46.984ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6882024-09-09T17:49:47.003ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6892024-09-09T17:49:47.003ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6902024-09-09T17:49:47.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6912024-09-09T17:49:47.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
6922024-09-09T17:49:47.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6932024-09-09T17:49:47.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6942024-09-09T17:49:47.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6952024-09-09T17:49:47.060ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6962024-09-09T17:49:47.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6972024-09-09T17:49:47.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
6982024-09-09T17:49:47.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6992024-09-09T17:49:47.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7002024-09-09T17:49:47.114ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7012024-09-09T17:49:47.114ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7022024-09-09T17:49:47.133ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7032024-09-09T17:49:47.133ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
7042024-09-09T17:49:47.133ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
7052024-09-09T17:49:47.133ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7062024-09-09T17:49:47.164ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
7072024-09-09T17:49:47.164ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
7082024-09-09T17:49:47.164ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
7092024-09-09T17:49:47.164ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
7102024-09-09T17:49:47.164ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
7112024-09-09T17:49:47.164ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 180437 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = a21eace3-fe11-4956-8072-ee1b060dff60 response_code = 500 uri = /omicron-zones
7122024-09-09T17:49:47.165ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a21eace3-fe11-4956-8072-ee1b060dff60", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a21eace3-fe11-4956-8072-ee1b060dff60" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 195.256725ms sled_address = [fd00:1122:3344:101::1]:12345
7132024-09-09T17:49:47.361ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7142024-09-09T17:49:47.362ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7152024-09-09T17:49:47.362ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7162024-09-09T17:49:47.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7172024-09-09T17:49:47.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7182024-09-09T17:49:47.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7192024-09-09T17:49:47.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
7202024-09-09T17:49:47.416ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7212024-09-09T17:49:47.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7222024-09-09T17:49:47.434ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7232024-09-09T17:49:47.434ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7242024-09-09T17:49:47.452ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7252024-09-09T17:49:47.452ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7262024-09-09T17:49:47.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7272024-09-09T17:49:47.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7282024-09-09T17:49:47.489ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7292024-09-09T17:49:47.489ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7302024-09-09T17:49:47.507ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7312024-09-09T17:49:47.507ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
7322024-09-09T17:49:47.507ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
7332024-09-09T17:49:47.508ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7342024-09-09T17:49:47.540ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
7352024-09-09T17:49:47.540ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
7362024-09-09T17:49:47.540ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
7372024-09-09T17:49:47.540ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
7382024-09-09T17:49:47.540ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
7392024-09-09T17:49:47.540ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 178305 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 73e568f2-2175-4bd8-a8c2-1446fd33e761 response_code = 500 uri = /omicron-zones
7402024-09-09T17:49:47.540ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "73e568f2-2175-4bd8-a8c2-1446fd33e761", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "73e568f2-2175-4bd8-a8c2-1446fd33e761" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 223.717741ms sled_address = [fd00:1122:3344:101::1]:12345
7412024-09-09T17:49:47.765ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7422024-09-09T17:49:47.765ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7432024-09-09T17:49:47.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7442024-09-09T17:49:47.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7452024-09-09T17:49:47.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7462024-09-09T17:49:47.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7472024-09-09T17:49:47.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
7482024-09-09T17:49:47.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7492024-09-09T17:49:47.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7502024-09-09T17:49:47.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7512024-09-09T17:49:47.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7522024-09-09T17:49:47.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7532024-09-09T17:49:47.855ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7542024-09-09T17:49:47.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7552024-09-09T17:49:47.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7562024-09-09T17:49:47.891ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7572024-09-09T17:49:47.891ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7582024-09-09T17:49:47.907ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7592024-09-09T17:49:47.907ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
7602024-09-09T17:49:47.907ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
7612024-09-09T17:49:47.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7622024-09-09T17:49:47.937ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
7632024-09-09T17:49:47.937ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
7642024-09-09T17:49:47.937ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
7652024-09-09T17:49:47.937ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
7662024-09-09T17:49:47.937ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
7672024-09-09T17:49:47.937ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 172013 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = cd93707a-2345-473e-8f2d-d8e4d2246e54 response_code = 500 uri = /omicron-zones
7682024-09-09T17:49:47.938ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cd93707a-2345-473e-8f2d-d8e4d2246e54", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cd93707a-2345-473e-8f2d-d8e4d2246e54" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 122.558609ms sled_address = [fd00:1122:3344:101::1]:12345
7692024-09-09T17:49:48.061ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7702024-09-09T17:49:48.061ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7712024-09-09T17:49:48.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7722024-09-09T17:49:48.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7732024-09-09T17:49:48.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7742024-09-09T17:49:48.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7752024-09-09T17:49:48.097ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
7762024-09-09T17:49:48.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7772024-09-09T17:49:48.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7782024-09-09T17:49:48.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7792024-09-09T17:49:48.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7802024-09-09T17:49:48.150ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7812024-09-09T17:49:48.150ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7822024-09-09T17:49:48.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7832024-09-09T17:49:48.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7842024-09-09T17:49:48.183ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7852024-09-09T17:49:48.183ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7862024-09-09T17:49:48.199ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7872024-09-09T17:49:48.199ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
7882024-09-09T17:49:48.200ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
7892024-09-09T17:49:48.200ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7902024-09-09T17:49:48.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
7912024-09-09T17:49:48.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
7922024-09-09T17:49:48.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
7932024-09-09T17:49:48.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
7942024-09-09T17:49:48.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
7952024-09-09T17:49:48.231ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 169535 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 9d9e39ec-beb2-4f30-92e8-120dd02dcb97 response_code = 500 uri = /omicron-zones
7962024-09-09T17:49:48.231ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9d9e39ec-beb2-4f30-92e8-120dd02dcb97", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9d9e39ec-beb2-4f30-92e8-120dd02dcb97" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 273.131022ms sled_address = [fd00:1122:3344:101::1]:12345
7972024-09-09T17:49:48.506ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7982024-09-09T17:49:48.507ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7992024-09-09T17:49:48.507ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8002024-09-09T17:49:48.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8012024-09-09T17:49:48.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8022024-09-09T17:49:48.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8032024-09-09T17:49:48.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
8042024-09-09T17:49:48.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8052024-09-09T17:49:48.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8062024-09-09T17:49:48.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8072024-09-09T17:49:48.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8082024-09-09T17:49:48.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8092024-09-09T17:49:48.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8102024-09-09T17:49:48.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8112024-09-09T17:49:48.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8122024-09-09T17:49:48.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8132024-09-09T17:49:48.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8142024-09-09T17:49:48.651ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8152024-09-09T17:49:48.651ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
8162024-09-09T17:49:48.651ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
8172024-09-09T17:49:48.652ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8182024-09-09T17:49:48.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
8192024-09-09T17:49:48.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
8202024-09-09T17:49:48.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
8212024-09-09T17:49:48.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
8222024-09-09T17:49:48.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
8232024-09-09T17:49:48.681ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 174434 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = e4b49af6-4222-4aed-8862-a0643b3fa692 response_code = 500 uri = /omicron-zones
8242024-09-09T17:49:48.682ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e4b49af6-4222-4aed-8862-a0643b3fa692", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e4b49af6-4222-4aed-8862-a0643b3fa692" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 184.610391ms sled_address = [fd00:1122:3344:101::1]:12345
8252024-09-09T17:49:48.867ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8262024-09-09T17:49:48.868ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8272024-09-09T17:49:48.868ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8282024-09-09T17:49:48.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8292024-09-09T17:49:48.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8302024-09-09T17:49:48.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8312024-09-09T17:49:48.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
8322024-09-09T17:49:48.922ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8332024-09-09T17:49:48.922ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8342024-09-09T17:49:48.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8352024-09-09T17:49:48.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8362024-09-09T17:49:48.958ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8372024-09-09T17:49:48.958ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8382024-09-09T17:49:48.976ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8392024-09-09T17:49:48.976ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8402024-09-09T17:49:48.994ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8412024-09-09T17:49:48.994ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8422024-09-09T17:49:49.010ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8432024-09-09T17:49:49.010ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
8442024-09-09T17:49:49.011ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
8452024-09-09T17:49:49.011ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8462024-09-09T17:49:49.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
8472024-09-09T17:49:49.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
8482024-09-09T17:49:49.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
8492024-09-09T17:49:49.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
8502024-09-09T17:49:49.041ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
8512024-09-09T17:49:49.041ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 173210 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 822efac4-43de-49ca-a32b-914cbbb4b688 response_code = 500 uri = /omicron-zones
8522024-09-09T17:49:49.041ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "822efac4-43de-49ca-a32b-914cbbb4b688", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "822efac4-43de-49ca-a32b-914cbbb4b688" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 320.533635ms sled_address = [fd00:1122:3344:101::1]:12345
8532024-09-09T17:49:49.364ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8542024-09-09T17:49:49.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8552024-09-09T17:49:49.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8562024-09-09T17:49:49.382ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8572024-09-09T17:49:49.382ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8582024-09-09T17:49:49.400ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8592024-09-09T17:49:49.400ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
8602024-09-09T17:49:49.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8612024-09-09T17:49:49.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8622024-09-09T17:49:49.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8632024-09-09T17:49:49.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8642024-09-09T17:49:49.453ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8652024-09-09T17:49:49.453ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8662024-09-09T17:49:49.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8672024-09-09T17:49:49.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8682024-09-09T17:49:49.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8692024-09-09T17:49:49.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8702024-09-09T17:49:49.505ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8712024-09-09T17:49:49.505ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
8722024-09-09T17:49:49.505ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
8732024-09-09T17:49:49.505ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8742024-09-09T17:49:49.536ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
8752024-09-09T17:49:49.536ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
8762024-09-09T17:49:49.536ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
8772024-09-09T17:49:49.536ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
8782024-09-09T17:49:49.536ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
8792024-09-09T17:49:49.536ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 171949 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 6a3d1d5d-8b17-47a4-a56f-b8bc0d91e49c response_code = 500 uri = /omicron-zones
8802024-09-09T17:49:49.536ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6a3d1d5d-8b17-47a4-a56f-b8bc0d91e49c", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6a3d1d5d-8b17-47a4-a56f-b8bc0d91e49c" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 380.449319ms sled_address = [fd00:1122:3344:101::1]:12345
8812024-09-09T17:49:49.841ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 344.417447ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
8822024-09-09T17:49:49.919ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8832024-09-09T17:49:49.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8842024-09-09T17:49:49.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8852024-09-09T17:49:49.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8862024-09-09T17:49:49.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8872024-09-09T17:49:49.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8882024-09-09T17:49:49.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
8892024-09-09T17:49:49.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8902024-09-09T17:49:49.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8912024-09-09T17:49:49.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8922024-09-09T17:49:49.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8932024-09-09T17:49:50.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8942024-09-09T17:49:50.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8952024-09-09T17:49:50.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8962024-09-09T17:49:50.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8972024-09-09T17:49:50.046ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8982024-09-09T17:49:50.046ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8992024-09-09T17:49:50.065ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9002024-09-09T17:49:50.065ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
9012024-09-09T17:49:50.065ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
9022024-09-09T17:49:50.065ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9032024-09-09T17:49:50.096ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
9042024-09-09T17:49:50.097ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
9052024-09-09T17:49:50.097ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
9062024-09-09T17:49:50.097ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
9072024-09-09T17:49:50.097ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
9082024-09-09T17:49:50.097ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 177141 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 5d13c058-fb90-490c-99e7-1466ff9f0b66 response_code = 500 uri = /omicron-zones
9092024-09-09T17:49:50.097ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5d13c058-fb90-490c-99e7-1466ff9f0b66", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5d13c058-fb90-490c-99e7-1466ff9f0b66" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 333.717325ms sled_address = [fd00:1122:3344:101::1]:12345
9102024-09-09T17:49:50.433ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9112024-09-09T17:49:50.433ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9122024-09-09T17:49:50.433ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9132024-09-09T17:49:50.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9142024-09-09T17:49:50.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9152024-09-09T17:49:50.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9162024-09-09T17:49:50.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
9172024-09-09T17:49:50.487ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9182024-09-09T17:49:50.487ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9192024-09-09T17:49:50.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9202024-09-09T17:49:50.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9212024-09-09T17:49:50.524ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9222024-09-09T17:49:50.524ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9232024-09-09T17:49:50.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9242024-09-09T17:49:50.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9252024-09-09T17:49:50.561ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9262024-09-09T17:49:50.561ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9272024-09-09T17:49:50.579ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9282024-09-09T17:49:50.579ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
9292024-09-09T17:49:50.579ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
9302024-09-09T17:49:50.579ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9312024-09-09T17:49:50.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
9322024-09-09T17:49:50.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
9332024-09-09T17:49:50.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
9342024-09-09T17:49:50.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
9352024-09-09T17:49:50.611ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
9362024-09-09T17:49:50.611ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 177386 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = cc1d7722-67aa-44bb-90c6-eb0b147f49c4 response_code = 500 uri = /omicron-zones
9372024-09-09T17:49:50.611ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cc1d7722-67aa-44bb-90c6-eb0b147f49c4", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cc1d7722-67aa-44bb-90c6-eb0b147f49c4" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 817.222426ms sled_address = [fd00:1122:3344:101::1]:12345
9382024-09-09T17:49:51.431ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9392024-09-09T17:49:51.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9402024-09-09T17:49:51.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9412024-09-09T17:49:51.453ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9422024-09-09T17:49:51.453ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9432024-09-09T17:49:51.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9442024-09-09T17:49:51.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
9452024-09-09T17:49:51.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9462024-09-09T17:49:51.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9472024-09-09T17:49:51.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9482024-09-09T17:49:51.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9492024-09-09T17:49:51.528ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9502024-09-09T17:49:51.528ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9512024-09-09T17:49:51.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9522024-09-09T17:49:51.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9532024-09-09T17:49:51.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9542024-09-09T17:49:51.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9552024-09-09T17:49:51.584ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9562024-09-09T17:49:51.584ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
9572024-09-09T17:49:51.585ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
9582024-09-09T17:49:51.585ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9592024-09-09T17:49:51.618ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
9602024-09-09T17:49:51.618ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
9612024-09-09T17:49:51.619ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
9622024-09-09T17:49:51.619ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
9632024-09-09T17:49:51.619ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
9642024-09-09T17:49:51.619ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186795 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 2f57b058-4994-4ad2-80c6-4291c857f2f6 response_code = 500 uri = /omicron-zones
9652024-09-09T17:49:51.619ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2f57b058-4994-4ad2-80c6-4291c857f2f6", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2f57b058-4994-4ad2-80c6-4291c857f2f6" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 646.173527ms sled_address = [fd00:1122:3344:101::1]:12345
9662024-09-09T17:49:52.268ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9672024-09-09T17:49:52.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9682024-09-09T17:49:52.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9692024-09-09T17:49:52.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9702024-09-09T17:49:52.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9712024-09-09T17:49:52.307ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9722024-09-09T17:49:52.307ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
9732024-09-09T17:49:52.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9742024-09-09T17:49:52.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9752024-09-09T17:49:52.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9762024-09-09T17:49:52.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9772024-09-09T17:49:52.362ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9782024-09-09T17:49:52.362ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9792024-09-09T17:49:52.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9802024-09-09T17:49:52.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9812024-09-09T17:49:52.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9822024-09-09T17:49:52.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9832024-09-09T17:49:52.417ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9842024-09-09T17:49:52.417ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
9852024-09-09T17:49:52.417ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
9862024-09-09T17:49:52.417ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9872024-09-09T17:49:52.449ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
9882024-09-09T17:49:52.449ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
9892024-09-09T17:49:52.449ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
9902024-09-09T17:49:52.449ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
9912024-09-09T17:49:52.449ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
9922024-09-09T17:49:52.449ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 180604 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 899c1886-41d8-45f4-9ea8-b37877c48799 response_code = 500 uri = /omicron-zones
9932024-09-09T17:49:52.449ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "899c1886-41d8-45f4-9ea8-b37877c48799", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "899c1886-41d8-45f4-9ea8-b37877c48799" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.248289798s sled_address = [fd00:1122:3344:101::1]:12345
9942024-09-09T17:49:53.702ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9952024-09-09T17:49:53.703ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9962024-09-09T17:49:53.703ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9972024-09-09T17:49:53.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9982024-09-09T17:49:53.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9992024-09-09T17:49:53.740ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10002024-09-09T17:49:53.740ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
10012024-09-09T17:49:53.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10022024-09-09T17:49:53.759ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10032024-09-09T17:49:53.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10042024-09-09T17:49:53.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10052024-09-09T17:49:53.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10062024-09-09T17:49:53.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10072024-09-09T17:49:53.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10082024-09-09T17:49:53.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10092024-09-09T17:49:53.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10102024-09-09T17:49:53.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10112024-09-09T17:49:53.850ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10122024-09-09T17:49:53.850ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
10132024-09-09T17:49:53.850ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
10142024-09-09T17:49:53.850ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10152024-09-09T17:49:53.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
10162024-09-09T17:49:53.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
10172024-09-09T17:49:53.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
10182024-09-09T17:49:53.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
10192024-09-09T17:49:53.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
10202024-09-09T17:49:53.882ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 179377 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 3e542c35-6478-443b-9d84-726782fb38f0 response_code = 500 uri = /omicron-zones
10212024-09-09T17:49:53.882ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3e542c35-6478-443b-9d84-726782fb38f0", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3e542c35-6478-443b-9d84-726782fb38f0" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 906.786587ms sled_address = [fd00:1122:3344:101::1]:12345
10222024-09-09T17:49:54.793ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10232024-09-09T17:49:54.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10242024-09-09T17:49:54.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10252024-09-09T17:49:54.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10262024-09-09T17:49:54.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10272024-09-09T17:49:54.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10282024-09-09T17:49:54.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
10292024-09-09T17:49:54.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10302024-09-09T17:49:54.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10312024-09-09T17:49:54.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10322024-09-09T17:49:54.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10332024-09-09T17:49:54.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10342024-09-09T17:49:54.881ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10352024-09-09T17:49:54.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10362024-09-09T17:49:54.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10372024-09-09T17:49:54.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10382024-09-09T17:49:54.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10392024-09-09T17:49:54.933ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10402024-09-09T17:49:54.934ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
10412024-09-09T17:49:54.934ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
10422024-09-09T17:49:54.934ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10432024-09-09T17:49:54.965ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
10442024-09-09T17:49:54.965ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
10452024-09-09T17:49:54.965ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
10462024-09-09T17:49:54.965ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
10472024-09-09T17:49:54.965ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
10482024-09-09T17:49:54.966ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 172392 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = d228329b-4227-492f-8583-1c485e80b25b response_code = 500 uri = /omicron-zones
10492024-09-09T17:49:54.966ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d228329b-4227-492f-8583-1c485e80b25b", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d228329b-4227-492f-8583-1c485e80b25b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.584378427s sled_address = [fd00:1122:3344:101::1]:12345
10502024-09-09T17:49:56.556ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10512024-09-09T17:49:56.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10522024-09-09T17:49:56.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10532024-09-09T17:49:56.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10542024-09-09T17:49:56.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10552024-09-09T17:49:56.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10562024-09-09T17:49:56.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
10572024-09-09T17:49:56.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10582024-09-09T17:49:56.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10592024-09-09T17:49:56.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10602024-09-09T17:49:56.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10612024-09-09T17:49:56.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10622024-09-09T17:49:56.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10632024-09-09T17:49:56.673ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10642024-09-09T17:49:56.673ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10652024-09-09T17:49:56.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10662024-09-09T17:49:56.693ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10672024-09-09T17:49:56.710ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10682024-09-09T17:49:56.711ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
10692024-09-09T17:49:56.711ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
10702024-09-09T17:49:56.711ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10712024-09-09T17:49:56.744ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
10722024-09-09T17:49:56.744ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
10732024-09-09T17:49:56.744ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
10742024-09-09T17:49:56.744ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
10752024-09-09T17:49:56.744ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
10762024-09-09T17:49:56.744ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 187384 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 579c16f2-f5d1-434d-95ff-13bcff371332 response_code = 500 uri = /omicron-zones
10772024-09-09T17:49:56.744ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "579c16f2-f5d1-434d-95ff-13bcff371332", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "579c16f2-f5d1-434d-95ff-13bcff371332" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.900136351s sled_address = [fd00:1122:3344:101::1]:12345
10782024-09-09T17:49:57.410ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59472
10792024-09-09T17:49:57.410ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59472 req_id = e31c054e-75a9-4ccf-8be7-15255c9bf1ef response_code = 200 uri = /baseboard
10802024-09-09T17:49:58.651ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10812024-09-09T17:49:58.651ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10822024-09-09T17:49:58.651ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10832024-09-09T17:49:58.672ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10842024-09-09T17:49:58.672ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10852024-09-09T17:49:58.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10862024-09-09T17:49:58.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
10872024-09-09T17:49:58.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10882024-09-09T17:49:58.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10892024-09-09T17:49:58.724ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10902024-09-09T17:49:58.724ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10912024-09-09T17:49:58.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10922024-09-09T17:49:58.742ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10932024-09-09T17:49:58.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10942024-09-09T17:49:58.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10952024-09-09T17:49:58.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10962024-09-09T17:49:58.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10972024-09-09T17:49:58.795ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10982024-09-09T17:49:58.795ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
10992024-09-09T17:49:58.795ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
11002024-09-09T17:49:58.795ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11012024-09-09T17:49:58.827ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
11022024-09-09T17:49:58.827ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
11032024-09-09T17:49:58.827ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
11042024-09-09T17:49:58.827ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
11052024-09-09T17:49:58.827ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
11062024-09-09T17:49:58.827ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 175508 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 74ac4d8c-71c8-427c-9195-111614685adc response_code = 500 uri = /omicron-zones
11072024-09-09T17:49:58.827ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "74ac4d8c-71c8-427c-9195-111614685adc", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "74ac4d8c-71c8-427c-9195-111614685adc" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 942.631949ms sled_address = [fd00:1122:3344:101::1]:12345
11082024-09-09T17:49:59.774ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11092024-09-09T17:49:59.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11102024-09-09T17:49:59.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11112024-09-09T17:49:59.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11122024-09-09T17:49:59.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11132024-09-09T17:49:59.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11142024-09-09T17:49:59.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
11152024-09-09T17:49:59.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11162024-09-09T17:49:59.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11172024-09-09T17:49:59.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11182024-09-09T17:49:59.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11192024-09-09T17:49:59.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11202024-09-09T17:49:59.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11212024-09-09T17:49:59.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11222024-09-09T17:49:59.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11232024-09-09T17:49:59.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11242024-09-09T17:49:59.903ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11252024-09-09T17:49:59.919ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11262024-09-09T17:49:59.919ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
11272024-09-09T17:49:59.920ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
11282024-09-09T17:49:59.920ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11292024-09-09T17:49:59.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
11302024-09-09T17:49:59.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
11312024-09-09T17:49:59.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
11322024-09-09T17:49:59.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
11332024-09-09T17:49:59.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
11342024-09-09T17:49:59.950ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 174813 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = a6d9d81a-52c0-497a-9afc-f94b1ecb46a5 response_code = 500 uri = /omicron-zones
11352024-09-09T17:49:59.950ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a6d9d81a-52c0-497a-9afc-f94b1ecb46a5", "content-length": "124", "date": "Mon, 09 Sep 2024 17:49:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a6d9d81a-52c0-497a-9afc-f94b1ecb46a5" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.966178175s sled_address = [fd00:1122:3344:101::1]:12345
11362024-09-09T17:50:01.923ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11372024-09-09T17:50:01.923ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11382024-09-09T17:50:01.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11392024-09-09T17:50:01.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11402024-09-09T17:50:01.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11412024-09-09T17:50:01.965ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11422024-09-09T17:50:01.965ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
11432024-09-09T17:50:01.983ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11442024-09-09T17:50:01.983ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11452024-09-09T17:50:02.001ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11462024-09-09T17:50:02.001ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11472024-09-09T17:50:02.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11482024-09-09T17:50:02.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11492024-09-09T17:50:02.041ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11502024-09-09T17:50:02.041ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11512024-09-09T17:50:02.059ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11522024-09-09T17:50:02.059ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11532024-09-09T17:50:02.077ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11542024-09-09T17:50:02.077ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
11552024-09-09T17:50:02.078ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
11562024-09-09T17:50:02.078ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11572024-09-09T17:50:02.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
11582024-09-09T17:50:02.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
11592024-09-09T17:50:02.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
11602024-09-09T17:50:02.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
11612024-09-09T17:50:02.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
11622024-09-09T17:50:02.109ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185197 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 882c6d71-1a1b-459b-a798-8a7a98404e5c response_code = 500 uri = /omicron-zones
11632024-09-09T17:50:02.109ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "882c6d71-1a1b-459b-a798-8a7a98404e5c", "content-length": "124", "date": "Mon, 09 Sep 2024 17:50:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "882c6d71-1a1b-459b-a798-8a7a98404e5c" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.305091316s sled_address = [fd00:1122:3344:101::1]:12345
11642024-09-09T17:50:04.422ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11652024-09-09T17:50:04.422ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11662024-09-09T17:50:04.422ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11672024-09-09T17:50:04.443ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11682024-09-09T17:50:04.443ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11692024-09-09T17:50:04.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11702024-09-09T17:50:04.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
11712024-09-09T17:50:04.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11722024-09-09T17:50:04.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11732024-09-09T17:50:04.500ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11742024-09-09T17:50:04.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11752024-09-09T17:50:04.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11762024-09-09T17:50:04.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11772024-09-09T17:50:04.537ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11782024-09-09T17:50:04.537ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11792024-09-09T17:50:04.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11802024-09-09T17:50:04.555ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11812024-09-09T17:50:04.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11822024-09-09T17:50:04.574ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
11832024-09-09T17:50:04.574ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
11842024-09-09T17:50:04.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11852024-09-09T17:50:04.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
11862024-09-09T17:50:04.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
11872024-09-09T17:50:04.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
11882024-09-09T17:50:04.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
11892024-09-09T17:50:04.606ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
11902024-09-09T17:50:04.606ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 183309 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 139838bc-39ab-43fb-91d1-68153d7a783f response_code = 500 uri = /omicron-zones
11912024-09-09T17:50:04.606ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "139838bc-39ab-43fb-91d1-68153d7a783f", "content-length": "124", "date": "Mon, 09 Sep 2024 17:50:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "139838bc-39ab-43fb-91d1-68153d7a783f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.733733282s sled_address = [fd00:1122:3344:101::1]:12345
11922024-09-09T17:50:05.263ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 498.746277ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
11932024-09-09T17:50:09.354ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11942024-09-09T17:50:09.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11952024-09-09T17:50:09.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11962024-09-09T17:50:09.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11972024-09-09T17:50:09.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11982024-09-09T17:50:09.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11992024-09-09T17:50:09.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
12002024-09-09T17:50:09.415ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12012024-09-09T17:50:09.415ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12022024-09-09T17:50:09.433ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12032024-09-09T17:50:09.433ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12042024-09-09T17:50:09.452ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12052024-09-09T17:50:09.452ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12062024-09-09T17:50:09.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12072024-09-09T17:50:09.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12082024-09-09T17:50:09.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12092024-09-09T17:50:09.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12102024-09-09T17:50:09.507ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12112024-09-09T17:50:09.507ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
12122024-09-09T17:50:09.507ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
12132024-09-09T17:50:09.508ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12142024-09-09T17:50:09.539ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
12152024-09-09T17:50:09.539ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
12162024-09-09T17:50:09.539ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
12172024-09-09T17:50:09.539ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
12182024-09-09T17:50:09.539ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
12192024-09-09T17:50:09.539ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 184532 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 2dc411ae-43ca-4819-8102-20d8686a0a99 response_code = 500 uri = /omicron-zones
12202024-09-09T17:50:09.539ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2dc411ae-43ca-4819-8102-20d8686a0a99", "content-length": "124", "date": "Mon, 09 Sep 2024 17:50:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2dc411ae-43ca-4819-8102-20d8686a0a99" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.583487947s sled_address = [fd00:1122:3344:101::1]:12345
12212024-09-09T17:50:14.137ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12222024-09-09T17:50:14.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12232024-09-09T17:50:14.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12242024-09-09T17:50:14.159ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12252024-09-09T17:50:14.159ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12262024-09-09T17:50:14.179ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12272024-09-09T17:50:14.179ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
12282024-09-09T17:50:14.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12292024-09-09T17:50:14.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12302024-09-09T17:50:14.217ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12312024-09-09T17:50:14.217ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12322024-09-09T17:50:14.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12332024-09-09T17:50:14.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12342024-09-09T17:50:14.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12352024-09-09T17:50:14.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12362024-09-09T17:50:14.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12372024-09-09T17:50:14.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12382024-09-09T17:50:14.290ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12392024-09-09T17:50:14.290ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
12402024-09-09T17:50:14.290ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
12412024-09-09T17:50:14.290ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12422024-09-09T17:50:14.323ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
12432024-09-09T17:50:14.323ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
12442024-09-09T17:50:14.323ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
12452024-09-09T17:50:14.323ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
12462024-09-09T17:50:14.323ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
12472024-09-09T17:50:14.323ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185472 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 5d05f671-c9b9-4df4-a361-407a4545f752 response_code = 500 uri = /omicron-zones
12482024-09-09T17:50:14.323ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5d05f671-c9b9-4df4-a361-407a4545f752", "content-length": "124", "date": "Mon, 09 Sep 2024 17:50:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5d05f671-c9b9-4df4-a361-407a4545f752" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 6.447033041s sled_address = [fd00:1122:3344:101::1]:12345
12492024-09-09T17:50:20.789ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12502024-09-09T17:50:20.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12512024-09-09T17:50:20.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12522024-09-09T17:50:20.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12532024-09-09T17:50:20.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12542024-09-09T17:50:20.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12552024-09-09T17:50:20.830ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
12562024-09-09T17:50:20.845ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 884.643346ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
12572024-09-09T17:50:20.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12582024-09-09T17:50:20.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12592024-09-09T17:50:20.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12602024-09-09T17:50:20.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12612024-09-09T17:50:20.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12622024-09-09T17:50:20.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12632024-09-09T17:50:20.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12642024-09-09T17:50:20.903ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12652024-09-09T17:50:20.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12662024-09-09T17:50:20.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12672024-09-09T17:50:20.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12682024-09-09T17:50:20.938ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
12692024-09-09T17:50:20.939ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
12702024-09-09T17:50:20.939ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12712024-09-09T17:50:20.972ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
12722024-09-09T17:50:20.972ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
12732024-09-09T17:50:20.972ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
12742024-09-09T17:50:20.972ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
12752024-09-09T17:50:20.972ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
12762024-09-09T17:50:20.972ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 182130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 2961c412-2375-428f-917d-18102a529c97 response_code = 500 uri = /omicron-zones
12772024-09-09T17:50:20.972ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2961c412-2375-428f-917d-18102a529c97", "content-length": "124", "date": "Mon, 09 Sep 2024 17:50:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2961c412-2375-428f-917d-18102a529c97" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 8.156944835s sled_address = [fd00:1122:3344:101::1]:12345
12782024-09-09T17:50:27.525ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55263
12792024-09-09T17:50:27.525ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55263 req_id = 3180a94a-9c10-4f11-aa66-7ed27f8944a5 response_code = 200 uri = /baseboard
12802024-09-09T17:50:29.153ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12812024-09-09T17:50:29.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12822024-09-09T17:50:29.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12832024-09-09T17:50:29.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12842024-09-09T17:50:29.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12852024-09-09T17:50:29.193ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12862024-09-09T17:50:29.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
12872024-09-09T17:50:29.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12882024-09-09T17:50:29.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12892024-09-09T17:50:29.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12902024-09-09T17:50:29.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12912024-09-09T17:50:29.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12922024-09-09T17:50:29.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12932024-09-09T17:50:29.266ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12942024-09-09T17:50:29.266ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12952024-09-09T17:50:29.284ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12962024-09-09T17:50:29.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12972024-09-09T17:50:29.302ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12982024-09-09T17:50:29.302ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
12992024-09-09T17:50:29.302ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
13002024-09-09T17:50:29.302ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13012024-09-09T17:50:29.335ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13022024-09-09T17:50:29.335ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13032024-09-09T17:50:29.335ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13042024-09-09T17:50:29.335ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13052024-09-09T17:50:29.335ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13062024-09-09T17:50:29.335ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2", TimeNotSynchronized), ("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b", TimeNotSynchronized), ("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b", TimeNotSynchronized), ("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554", TimeNotSynchronized), ("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 181356 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 650bc940-ec8f-4037-beb4-7ae648dfa264 response_code = 500 uri = /omicron-zones
13072024-09-09T17:50:29.335ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "650bc940-ec8f-4037-beb4-7ae648dfa264", "content-length": "124", "date": "Mon, 09 Sep 2024 17:50:29 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "650bc940-ec8f-4037-beb4-7ae648dfa264" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 9.211943116s sled_address = [fd00:1122:3344:101::1]:12345
13082024-09-09T17:50:36.812ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.83086631s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
13092024-09-09T17:50:38.574ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13102024-09-09T17:50:38.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13112024-09-09T17:50:38.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13122024-09-09T17:50:38.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13132024-09-09T17:50:38.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13142024-09-09T17:50:38.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13152024-09-09T17:50:38.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
13162024-09-09T17:50:38.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13172024-09-09T17:50:38.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13182024-09-09T17:50:38.655ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13192024-09-09T17:50:38.655ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13202024-09-09T17:50:38.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13212024-09-09T17:50:38.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
13222024-09-09T17:50:38.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13232024-09-09T17:50:38.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13242024-09-09T17:50:38.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13252024-09-09T17:50:38.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13262024-09-09T17:50:38.729ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13272024-09-09T17:50:38.729ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
13282024-09-09T17:50:38.730ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
13292024-09-09T17:50:38.730ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13302024-09-09T17:50:38.814ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13312024-09-09T17:50:38.814ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13322024-09-09T17:50:38.814ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13332024-09-09T17:50:38.814ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13342024-09-09T17:50:38.814ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13352024-09-09T17:50:38.906ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b file = illumos-utils/src/zone.rs:316
13362024-09-09T17:50:38.910ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b file = illumos-utils/src/zone.rs:316
13372024-09-09T17:50:38.914ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48 file = illumos-utils/src/zone.rs:316
13382024-09-09T17:50:38.919ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2 file = illumos-utils/src/zone.rs:316
13392024-09-09T17:50:38.923ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554 file = illumos-utils/src/zone.rs:316
13402024-09-09T17:50:38.928ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b file = illumos-utils/src/zone.rs:354
13412024-09-09T17:50:38.933ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b file = illumos-utils/src/zone.rs:354
13422024-09-09T17:50:38.938ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48 file = illumos-utils/src/zone.rs:354
13432024-09-09T17:50:38.942ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2 file = illumos-utils/src/zone.rs:354
13442024-09-09T17:50:38.947ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554 file = illumos-utils/src/zone.rs:354
13452024-09-09T17:50:41.932ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='67aa186b-3457-4f4c-a272-6d6f7b31af3b'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13462024-09-09T17:50:41.932ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13472024-09-09T17:50:41.938ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='f48d5fbc-b227-4070-9b8c-1daa5dd4af48'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13482024-09-09T17:50:41.938ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13492024-09-09T17:50:41.965ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='14dd30ae-7b79-425c-93c7-fe02960c834b'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13502024-09-09T17:50:41.965ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13512024-09-09T17:50:41.996ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3409b0d3-ece0-4389-9b68-d2f49d8e8554'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13522024-09-09T17:50:41.997ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13532024-09-09T17:50:42.003ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='83a77c6e-f99a-4321-9983-b522202ce9a2'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13542024-09-09T17:50:42.004ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13552024-09-09T17:50:42.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 49.302509ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13562024-09-09T17:50:42.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 57.815922ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13572024-09-09T17:50:42.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 70.24075ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13582024-09-09T17:50:42.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 48.638587ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13592024-09-09T17:50:42.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 116.821199ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13602024-09-09T17:50:42.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 25.634824ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13612024-09-09T17:50:42.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 87.631845ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13622024-09-09T17:50:42.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 94.850453ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13632024-09-09T17:50:42.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 147.909561ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13642024-09-09T17:50:42.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 101.605271ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13652024-09-09T17:50:42.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 66.996066ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13662024-09-09T17:50:43.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 79.269717ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13672024-09-09T17:50:43.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 245.523866ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13682024-09-09T17:50:43.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 434.115625ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13692024-09-09T17:50:43.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 496.020062ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13702024-09-09T17:50:43.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 249.36093ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13712024-09-09T17:50:43.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 167.748379ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13722024-09-09T17:50:43.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 295.279012ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13732024-09-09T17:50:44.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 1.058247901s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13742024-09-09T17:50:44.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 1.033585426s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13752024-09-09T17:50:44.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 547.820702ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13762024-09-09T17:50:45.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 561.029504ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13772024-09-09T17:50:45.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 598.694597ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13782024-09-09T17:50:45.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 476.109288ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13792024-09-09T17:50:45.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 730.117125ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13802024-09-09T17:50:45.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 658.887503ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13812024-09-09T17:50:45.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 507.773353ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13822024-09-09T17:50:45.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 895.965157ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13832024-09-09T17:50:46.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.26380491s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13842024-09-09T17:50:46.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 693.915916ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13852024-09-09T17:50:46.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 595.20053ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13862024-09-09T17:50:46.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 653.616286ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13872024-09-09T17:50:46.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 920.926533ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13882024-09-09T17:50:46.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 756.052837ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13892024-09-09T17:50:46.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 1.155622191s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13902024-09-09T17:50:47.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 1.040455682s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13912024-09-09T17:50:47.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 590.525681ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13922024-09-09T17:50:47.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.194732639s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
13932024-09-09T17:50:47.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 1.259831144s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
13942024-09-09T17:50:47.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 989.985282ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13952024-09-09T17:50:48.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 1.317840101s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
13962024-09-09T17:50:48.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 717.848738ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13972024-09-09T17:50:48.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 836.154703ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
13982024-09-09T17:50:48.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.457735533s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
13992024-09-09T17:50:48.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 774.478371ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14002024-09-09T17:50:49.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 1.075180221s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14012024-09-09T17:50:49.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 1.145571407s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14022024-09-09T17:50:49.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 1.084367419s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
14032024-09-09T17:50:49.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.400509048s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14042024-09-09T17:50:50.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 823.405027ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14052024-09-09T17:50:50.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 927.473468ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14062024-09-09T17:50:50.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 812.430528ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14072024-09-09T17:50:50.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b") failed: Property not found. retry in 1.385335453s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
14082024-09-09T17:50:51.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 976.717814ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14092024-09-09T17:50:51.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.169391537s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14102024-09-09T17:50:51.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.068963422s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14112024-09-09T17:50:51.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 554.951379ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14122024-09-09T17:50:51.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 1.481126633s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14132024-09-09T17:50:52.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 1.375901808s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14142024-09-09T17:50:52.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.069090324s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14152024-09-09T17:50:52.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.174138228s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14162024-09-09T17:50:53.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 882.840439ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14172024-09-09T17:50:53.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.243447409s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14182024-09-09T17:50:53.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 774.879477ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14192024-09-09T17:50:53.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 662.216567ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14202024-09-09T17:50:53.743ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.833768988s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
14212024-09-09T17:50:54.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 928.246131ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14222024-09-09T17:50:54.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 940.306282ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14232024-09-09T17:50:54.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 1.227046159s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14242024-09-09T17:50:54.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.447762007s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14252024-09-09T17:50:55.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 1.097268416s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14262024-09-09T17:50:55.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.315284023s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14272024-09-09T17:50:55.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 749.466829ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14282024-09-09T17:50:56.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 757.105718ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14292024-09-09T17:50:56.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 535.197718ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14302024-09-09T17:50:56.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 908.341563ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14312024-09-09T17:50:56.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 851.903035ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14322024-09-09T17:50:56.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 1.223482145s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14332024-09-09T17:50:56.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.396334715s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14342024-09-09T17:50:57.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 1.499320394s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14352024-09-09T17:50:57.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 950.83993ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14362024-09-09T17:50:57.630ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58260
14372024-09-09T17:50:57.631ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58260 req_id = 474812b0-cef7-4b88-b849-1727e6a999ab response_code = 200 uri = /baseboard
14382024-09-09T17:50:58.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 975.241872ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14392024-09-09T17:50:58.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.081739443s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14402024-09-09T17:50:58.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.09370894s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14412024-09-09T17:50:58.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 924.373409ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14422024-09-09T17:50:59.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 1.449416437s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14432024-09-09T17:50:59.398ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.366710314s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14442024-09-09T17:50:59.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 507.345427ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14452024-09-09T17:50:59.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 1.428033894s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14462024-09-09T17:50:59.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.456986828s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14472024-09-09T17:51:00.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 577.866251ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14482024-09-09T17:51:00.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b") failed: Property not found. retry in 1.01290118s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14492024-09-09T17:51:01.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48") failed: Property not found. retry in 616.289914ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14502024-09-09T17:51:01.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554") failed: Property not found. retry in 785.083018ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14512024-09-09T17:51:01.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2") failed: Property not found. retry in 1.14768946s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14522024-09-09T17:51:02.634ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_14dd30ae-7b79-425c-93c7-fe02960c834b
14532024-09-09T17:51:02.634ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_67aa186b-3457-4f4c-a272-6d6f7b31af3b
14542024-09-09T17:51:02.634ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_f48d5fbc-b227-4070-9b8c-1daa5dd4af48
14552024-09-09T17:51:02.634ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_83a77c6e-f99a-4321-9983-b522202ce9a2
14562024-09-09T17:51:02.634ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_3409b0d3-ece0-4389-9b68-d2f49d8e8554
14572024-09-09T17:51:02.634ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
14582024-09-09T17:51:02.635ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
14592024-09-09T17:51:02.635ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 24021736 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36776 req_id = 1a291c2b-714a-48d0-9876-3b61c3a0d195 response_code = 204 uri = /omicron-zones
14602024-09-09T17:51:02.674ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52137
14612024-09-09T17:51:02.711ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::4]:32221 file = sled-agent/src/services.rs:3686
14622024-09-09T17:51:15.587ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3704
14632024-09-09T17:51:27.370ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3727
14642024-09-09T17:51:27.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 24696063 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:52137 req_id = 10250563-7473-42e7-a9f0-35232a0931e7 response_code = 204 uri = /cockroachdb
14652024-09-09T17:51:27.378ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.321320575s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
14662024-09-09T17:51:27.412ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 21f5073c-362d-497d-ae9e-f6ab85374703, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 30c18989-6f47-40e1-ab15-11c78531e2ea, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 18779e04-84b0-48e6-a8b5-146b8922c987, kind: Service { id: 30c18989-6f47-40e1-ab15-11c78531e2ea }, name: Name("ntp-30c18989-6f47-40e1-ab15-11c78531e2ea"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 220, 173])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 14dd30ae-7b79-425c-93c7-fe02960c834b, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83a77c6e-f99a-4321-9983-b522202ce9a2, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9711344c-11c4-490b-9560-e9c007eb0cf3, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 07c1bbdd-f080-4837-9528-41234be47bda, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: ca022d43-3da4-4507-941c-6fd9da5b668d, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: fd75f244-5e91-4f63-b3b2-ae4139e91823, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: fe9e311d-41c3-4a6e-8e3e-3fbb54dee639, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: e017ec87-832c-4092-8516-57b23da956bc, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: ef3a2fd8-3786-4198-8c2d-a9442073f2de, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: bebbab47-f19c-4831-a25e-344318d56393, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 4a391a53-62d4-4c58-a8df-837ca0cab6d9, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 88cec213-8d41-4e46-aea3-2d037677a065, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2ce49a1d-0f2c-40bf-9a41-d49a169c3abb, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External } } } }, OmicronZoneConfig { id: a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External } } } }, OmicronZoneConfig { id: c52a5621-100d-4082-b483-9a9b71016544, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External } } } }, OmicronZoneConfig { id: e37b75a6-6c18-4702-a94d-ecb36cc45c07, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 18d54208-fbd9-4a9a-bb4c-0dda56eeac06, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d731515-3696-45fe-816a-2c2bf0393b4f, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool), kind: External } } } }, OmicronZoneConfig { id: a4f04eb1-0d05-41b6-b3d2-1d063e4e900e, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool), kind: External } } } }, OmicronZoneConfig { id: 899a683c-bd60-4b15-b12a-e5db4452f148, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14672024-09-09T17:51:27.412ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59416
14682024-09-09T17:51:27.414ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14692024-09-09T17:51:27.414ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 42b0964a-b1a3-443e-b006-c18e1ad8d003, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14702024-09-09T17:51:27.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14712024-09-09T17:51:27.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9704e6fa-a599-4b20-92b9-bc19ae282ee4, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14722024-09-09T17:51:27.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14732024-09-09T17:51:27.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 21f5073c-362d-497d-ae9e-f6ab85374703, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: InternalDns }, responder: ... }
14742024-09-09T17:51:27.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14752024-09-09T17:51:27.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f48d5fbc-b227-4070-9b8c-1daa5dd4af48, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14762024-09-09T17:51:27.502ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14772024-09-09T17:51:27.502ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 14dd30ae-7b79-425c-93c7-fe02960c834b, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14782024-09-09T17:51:27.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14792024-09-09T17:51:27.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3409b0d3-ece0-4389-9b68-d2f49d8e8554, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Cockroach }, responder: ... }
14802024-09-09T17:51:27.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14812024-09-09T17:51:27.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 67aa186b-3457-4f4c-a272-6d6f7b31af3b, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14822024-09-09T17:51:27.560ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14832024-09-09T17:51:27.560ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 83a77c6e-f99a-4321-9983-b522202ce9a2, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14842024-09-09T17:51:27.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9711344c-11c4-490b-9560-e9c007eb0cf3, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14852024-09-09T17:51:27.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9711344c-11c4-490b-9560-e9c007eb0cf3, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14862024-09-09T17:51:27.669ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:50551
14872024-09-09T17:51:27.669ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 513 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:50551 req_id = af334c33-f0c7-4ca4-9d8b-9c9ac6ee8dd3 response_code = 200 uri = /baseboard
14882024-09-09T17:51:27.705ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07c1bbdd-f080-4837-9528-41234be47bda, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14892024-09-09T17:51:27.705ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 07c1bbdd-f080-4837-9528-41234be47bda, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14902024-09-09T17:51:27.829ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e017ec87-832c-4092-8516-57b23da956bc, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
14912024-09-09T17:51:27.829ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e017ec87-832c-4092-8516-57b23da956bc, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14922024-09-09T17:51:27.966ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 88cec213-8d41-4e46-aea3-2d037677a065, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14932024-09-09T17:51:27.966ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 88cec213-8d41-4e46-aea3-2d037677a065, dataset_name: DatasetName { pool_name: ZpoolName { id: e68e093e-0654-4278-a79f-9c234ab4cc35 (zpool), kind: External }, kind: Crucible }, responder: ... }
14942024-09-09T17:51:28.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ce49a1d-0f2c-40bf-9a41-d49a169c3abb, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14952024-09-09T17:51:28.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2ce49a1d-0f2c-40bf-9a41-d49a169c3abb, dataset_name: DatasetName { pool_name: ZpoolName { id: f5eda459-d1e9-44f5-b731-f6490e3fc719 (zpool), kind: External }, kind: Crucible }, responder: ... }
14962024-09-09T17:51:28.238ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14972024-09-09T17:51:28.238ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03, dataset_name: DatasetName { pool_name: ZpoolName { id: 677df3fa-a1f5-4e63-93ee-592a272d5e3f (zpool), kind: External }, kind: Crucible }, responder: ... }
14982024-09-09T17:51:28.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c52a5621-100d-4082-b483-9a9b71016544, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
14992024-09-09T17:51:28.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c52a5621-100d-4082-b483-9a9b71016544, dataset_name: DatasetName { pool_name: ZpoolName { id: 89ffd24b-1dd8-4f52-85d7-6f3c66fce582 (zpool), kind: External }, kind: Crucible }, responder: ... }
15002024-09-09T17:51:28.478ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e37b75a6-6c18-4702-a94d-ecb36cc45c07, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15012024-09-09T17:51:28.478ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: e37b75a6-6c18-4702-a94d-ecb36cc45c07, dataset_name: DatasetName { pool_name: ZpoolName { id: 46b15cce-a870-48ea-aab8-5f90f01b99e4 (zpool), kind: External }, kind: Crucible }, responder: ... }
15022024-09-09T17:51:28.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18d54208-fbd9-4a9a-bb4c-0dda56eeac06, dataset_name: DatasetName { pool_name: ZpoolName { id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15032024-09-09T17:51:28.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 18d54208-fbd9-4a9a-bb4c-0dda56eeac06, dataset_name: DatasetName { pool_name: ZpoolName { id: 0720cae8-ac21-4fb2-9b27-8f836e4c5c41 (zpool), kind: External }, kind: Crucible }, responder: ... }
15042024-09-09T17:51:28.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d731515-3696-45fe-816a-2c2bf0393b4f, dataset_name: DatasetName { pool_name: ZpoolName { id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15052024-09-09T17:51:28.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0d731515-3696-45fe-816a-2c2bf0393b4f, dataset_name: DatasetName { pool_name: ZpoolName { id: f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299 (zpool), kind: External }, kind: Crucible }, responder: ... }
15062024-09-09T17:51:28.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a4f04eb1-0d05-41b6-b3d2-1d063e4e900e, dataset_name: DatasetName { pool_name: ZpoolName { id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15072024-09-09T17:51:28.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a4f04eb1-0d05-41b6-b3d2-1d063e4e900e, dataset_name: DatasetName { pool_name: ZpoolName { id: 2c079939-1543-4f7b-ab38-d421cbf95e07 (zpool), kind: External }, kind: Crucible }, responder: ... }
15082024-09-09T17:51:28.992ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 899a683c-bd60-4b15-b12a-e5db4452f148, dataset_name: DatasetName { pool_name: ZpoolName { id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
15092024-09-09T17:51:28.992ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 899a683c-bd60-4b15-b12a-e5db4452f148, dataset_name: DatasetName { pool_name: ZpoolName { id: b6fbc8d5-08ec-4cd5-ae5d-395890772649 (zpool), kind: External }, kind: Crucible }, responder: ... }
15102024-09-09T17:51:29.106ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15112024-09-09T17:51:29.107ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
15122024-09-09T17:51:29.107ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
15132024-09-09T17:51:29.107ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15142024-09-09T17:51:29.253ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
15152024-09-09T17:51:29.267ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
15162024-09-09T17:51:29.267ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15172024-09-09T17:51:29.281ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
15182024-09-09T17:51:29.281ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15192024-09-09T17:51:29.281ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15202024-09-09T17:51:29.305ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
15212024-09-09T17:51:29.306ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15222024-09-09T17:51:29.330ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15232024-09-09T17:51:29.330ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15242024-09-09T17:51:29.470ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
15252024-09-09T17:51:29.470ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15262024-09-09T17:51:29.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15272024-09-09T17:51:29.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15282024-09-09T17:51:29.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15292024-09-09T17:51:29.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15302024-09-09T17:51:29.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15312024-09-09T17:51:29.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15322024-09-09T17:51:29.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15332024-09-09T17:51:29.521ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148 file = illumos-utils/src/zone.rs:316
15342024-09-09T17:51:29.528ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639 file = illumos-utils/src/zone.rs:316
15352024-09-09T17:51:29.534ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544 file = illumos-utils/src/zone.rs:316
15362024-09-09T17:51:29.541ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07 file = illumos-utils/src/zone.rs:316
15372024-09-09T17:51:29.547ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc file = illumos-utils/src/zone.rs:316
15382024-09-09T17:51:29.553ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06 file = illumos-utils/src/zone.rs:316
15392024-09-09T17:51:29.713ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148 file = illumos-utils/src/zone.rs:354
15402024-09-09T17:51:29.718ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639 file = illumos-utils/src/zone.rs:354
15412024-09-09T17:51:29.723ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544 file = illumos-utils/src/zone.rs:354
15422024-09-09T17:51:29.727ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07 file = illumos-utils/src/zone.rs:354
15432024-09-09T17:51:29.731ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc file = illumos-utils/src/zone.rs:354
15442024-09-09T17:51:29.736ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06 file = illumos-utils/src/zone.rs:354
15452024-09-09T17:51:29.741ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f file = illumos-utils/src/zone.rs:316
15462024-09-09T17:51:29.746ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9 file = illumos-utils/src/zone.rs:316
15472024-09-09T17:51:29.751ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e file = illumos-utils/src/zone.rs:316
15482024-09-09T17:51:29.755ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393 file = illumos-utils/src/zone.rs:316
15492024-09-09T17:51:29.760ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065 file = illumos-utils/src/zone.rs:316
15502024-09-09T17:51:29.765ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de file = illumos-utils/src/zone.rs:316
15512024-09-09T17:51:29.771ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb file = illumos-utils/src/zone.rs:316
15522024-09-09T17:51:29.776ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03 file = illumos-utils/src/zone.rs:316
15532024-09-09T17:51:29.783ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f file = illumos-utils/src/zone.rs:354
15542024-09-09T17:51:29.826ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
15552024-09-09T17:51:29.826ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9 file = illumos-utils/src/zone.rs:354
15562024-09-09T17:51:29.832ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e file = illumos-utils/src/zone.rs:354
15572024-09-09T17:51:29.838ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393 file = illumos-utils/src/zone.rs:354
15582024-09-09T17:51:29.842ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065 file = illumos-utils/src/zone.rs:354
15592024-09-09T17:51:29.847ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de file = illumos-utils/src/zone.rs:354
15602024-09-09T17:51:29.852ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb file = illumos-utils/src/zone.rs:354
15612024-09-09T17:51:29.858ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03 file = illumos-utils/src/zone.rs:354
15622024-09-09T17:51:29.900ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
15632024-09-09T17:51:29.965ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
15642024-09-09T17:51:30.020ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
15652024-09-09T17:51:30.110ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
15662024-09-09T17:51:30.114ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
15672024-09-09T17:51:30.114ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
15682024-09-09T17:51:30.169ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 187, 44]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
15692024-09-09T17:51:30.169ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
15702024-09-09T17:51:30.170ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
15712024-09-09T17:51:30.170ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
15722024-09-09T17:51:30.256ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 223, 94]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
15732024-09-09T17:51:30.256ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
15742024-09-09T17:51:30.257ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
15752024-09-09T17:51:30.257ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
15762024-09-09T17:51:30.314ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 139, 64]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
15772024-09-09T17:51:30.314ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
15782024-09-09T17:51:30.315ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
15792024-09-09T17:51:30.315ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
15802024-09-09T17:51:30.383ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 185, 185]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
15812024-09-09T17:51:30.383ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
15822024-09-09T17:51:30.384ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
15832024-09-09T17:51:30.384ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
15842024-09-09T17:51:30.441ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 201, 216]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
15852024-09-09T17:51:30.441ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
15862024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
15872024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
15882024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
15892024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
15902024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
15912024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
15922024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
15932024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
15942024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
15952024-09-09T17:51:30.446ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
15962024-09-09T17:51:30.450ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
15972024-09-09T17:51:30.451ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15982024-09-09T17:51:30.505ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
15992024-09-09T17:51:30.508ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
16002024-09-09T17:51:30.511ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
16012024-09-09T17:51:30.512ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16022024-09-09T17:51:30.512ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16032024-09-09T17:51:30.515ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
16042024-09-09T17:51:30.518ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16052024-09-09T17:51:30.518ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
16062024-09-09T17:51:30.742ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8 file = illumos-utils/src/zone.rs:316
16072024-09-09T17:51:30.754ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda file = illumos-utils/src/zone.rs:316
16082024-09-09T17:51:30.766ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823 file = illumos-utils/src/zone.rs:316
16092024-09-09T17:51:30.779ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d file = illumos-utils/src/zone.rs:316
16102024-09-09T17:51:30.793ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3 file = illumos-utils/src/zone.rs:316
16112024-09-09T17:51:30.807ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8 file = illumos-utils/src/zone.rs:354
16122024-09-09T17:51:30.815ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda file = illumos-utils/src/zone.rs:354
16132024-09-09T17:51:30.826ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823 file = illumos-utils/src/zone.rs:354
16142024-09-09T17:51:30.837ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d file = illumos-utils/src/zone.rs:354
16152024-09-09T17:51:30.849ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3 file = illumos-utils/src/zone.rs:354
16162024-09-09T17:51:33.396ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16172024-09-09T17:51:33.396ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16182024-09-09T17:51:33.396ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
16192024-09-09T17:51:33.423ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
16202024-09-09T17:51:33.429ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16212024-09-09T17:51:33.430ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
16222024-09-09T17:51:33.477ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='9b58f32a-c07c-4cfa-87c1-9b2871ed0d30'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16232024-09-09T17:51:33.477ZINFOSledAgent (ServiceManager): Booting oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
16242024-09-09T17:51:33.485ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_2c079939-1543-4f7b-ab38-d421cbf95e07/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='2ea5dda5-9c0a-4c5b-955e-3c5837723ae3'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16252024-09-09T17:51:33.485ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_677df3fa-a1f5-4e63-93ee-592a272d5e3f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d0c17537-d1a2-4409-833a-33aa23ef83f4'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16262024-09-09T17:51:33.485ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_b6fbc8d5-08ec-4cd5-ae5d-395890772649/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='e3f15c64-b2e3-4a9c-8336-c0b3c9505c16'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16272024-09-09T17:51:33.485ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='53406b79-0976-4a48-8280-d3029d4e6872'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16282024-09-09T17:51:33.485ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_46b15cce-a870-48ea-aab8-5f90f01b99e4/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='55964865-3fa1-4d9a-b24d-58ae47545af2'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16292024-09-09T17:51:33.486ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_0720cae8-ac21-4fb2-9b27-8f836e4c5c41/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='436fe0e2-fa25-4c16-b0eb-46e7f8629d1e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16302024-09-09T17:51:33.486ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
16312024-09-09T17:51:33.493ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
16322024-09-09T17:51:33.498ZINFOSledAgent (ServiceManager): Booting oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
16332024-09-09T17:51:33.504ZINFOSledAgent (ServiceManager): Booting oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
16342024-09-09T17:51:33.510ZINFOSledAgent (ServiceManager): Booting oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
16352024-09-09T17:51:33.516ZINFOSledAgent (ServiceManager): Booting oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
16362024-09-09T17:51:33.538ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e68e093e-0654-4278-a79f-9c234ab4cc35/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3ca21a37-79fa-4bc4-98b0-5fcdd83fe07c'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16372024-09-09T17:51:33.538ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f5eda459-d1e9-44f5-b731-f6490e3fc719/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3a30973f-9726-4108-93d7-be71cb0864aa'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16382024-09-09T17:51:33.538ZINFOSledAgent (ServiceManager): Booting oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
16392024-09-09T17:51:33.545ZINFOSledAgent (ServiceManager): Booting oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
16402024-09-09T17:51:33.560ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='fe9e311d-41c3-4a6e-8e3e-3fbb54dee639'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16412024-09-09T17:51:33.561ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
16422024-09-09T17:51:33.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 53.145241ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
16432024-09-09T17:51:33.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 52.441238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
16442024-09-09T17:51:33.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 58.965826ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
16452024-09-09T17:51:33.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 74.993893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
16462024-09-09T17:51:33.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 46.579608ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
16472024-09-09T17:51:33.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 64.686543ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
16482024-09-09T17:51:33.937ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 28.413728ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
16492024-09-09T17:51:33.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 46.894415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
16502024-09-09T17:51:33.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 32.25378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
16512024-09-09T17:51:34.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 25.355303ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
16522024-09-09T17:51:34.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 41.70191ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
16532024-09-09T17:51:34.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 118.640259ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
16542024-09-09T17:51:34.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 69.277483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
16552024-09-09T17:51:35.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 72.122894ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
16562024-09-09T17:51:35.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 144.326285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
16572024-09-09T17:51:35.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 60.592723ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
16582024-09-09T17:51:35.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 54.44899ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
16592024-09-09T17:51:35.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 146.398689ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
16602024-09-09T17:51:35.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 119.382296ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
16612024-09-09T17:51:35.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 113.166299ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
16622024-09-09T17:51:35.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 101.18497ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
16632024-09-09T17:51:35.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 103.188356ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
16642024-09-09T17:51:35.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 90.031172ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
16652024-09-09T17:51:35.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 74.137032ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
16662024-09-09T17:51:35.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 123.537519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
16672024-09-09T17:51:35.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 96.483482ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
16682024-09-09T17:51:35.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 99.769987ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
16692024-09-09T17:51:35.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 279.902945ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
16702024-09-09T17:51:35.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 287.359228ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
16712024-09-09T17:51:35.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 162.087464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
16722024-09-09T17:51:35.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 222.677075ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
16732024-09-09T17:51:35.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 299.410149ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
16742024-09-09T17:51:35.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 119.531106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
16752024-09-09T17:51:35.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 292.636063ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
16762024-09-09T17:51:35.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 190.158483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
16772024-09-09T17:51:36.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 217.744599ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
16782024-09-09T17:51:36.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 199.828304ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
16792024-09-09T17:51:36.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 548.962575ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
16802024-09-09T17:51:36.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 138.589592ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
16812024-09-09T17:51:36.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 165.506183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
16822024-09-09T17:51:36.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 522.89946ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
16832024-09-09T17:51:36.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 303.767423ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
16842024-09-09T17:51:36.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 540.915104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
16852024-09-09T17:51:36.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 379.04853ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
16862024-09-09T17:51:36.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 511.367552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
16872024-09-09T17:51:36.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 347.080948ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
16882024-09-09T17:51:36.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 480.38722ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
16892024-09-09T17:51:36.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 597.869753ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
16902024-09-09T17:51:36.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 506.265386ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
16912024-09-09T17:51:36.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 252.64454ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
16922024-09-09T17:51:36.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 431.615666ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
16932024-09-09T17:51:36.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 250.507262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
16942024-09-09T17:51:36.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 842.790387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
16952024-09-09T17:51:36.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 419.684878ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
16962024-09-09T17:51:36.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 425.721169ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
16972024-09-09T17:51:36.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 886.320145ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
16982024-09-09T17:51:36.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 879.526991ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
16992024-09-09T17:51:36.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.065567352s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
17002024-09-09T17:51:36.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 657.864053ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
17012024-09-09T17:51:36.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.099289644s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
17022024-09-09T17:51:36.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 594.6043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
17032024-09-09T17:51:36.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 549.226871ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
17042024-09-09T17:51:36.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 882.894581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
17052024-09-09T17:51:36.972ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.057640786s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
17062024-09-09T17:51:36.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 753.64198ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
17072024-09-09T17:51:37.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 720.843828ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
17082024-09-09T17:51:37.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 806.781405ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
17092024-09-09T17:51:37.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.110461696s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
17102024-09-09T17:51:37.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 801.267618ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
17112024-09-09T17:51:37.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 648.235367ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
17122024-09-09T17:51:37.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.447115968s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
17132024-09-09T17:51:37.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.372580481s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
17142024-09-09T17:51:37.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.120238751s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
17152024-09-09T17:51:37.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.037362126s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
17162024-09-09T17:51:37.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.433609005s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
17172024-09-09T17:51:37.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.433804416s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
17182024-09-09T17:51:37.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.302632922s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
17192024-09-09T17:51:37.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.273830328s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
17202024-09-09T17:51:38.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.399311863s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
17212024-09-09T17:51:38.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 991.148017ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
17222024-09-09T17:51:38.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 852.186501ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
17232024-09-09T17:51:38.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 678.753556ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
17242024-09-09T17:51:38.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 856.572897ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
17252024-09-09T17:51:38.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 668.252858ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
17262024-09-09T17:51:38.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 771.015324ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
17272024-09-09T17:51:38.798ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17282024-09-09T17:51:38.798ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17292024-09-09T17:51:38.799ZINFOSledAgent (ServiceManager): Booting oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17302024-09-09T17:51:38.807ZINFOSledAgent (ServiceManager): Booting oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
17312024-09-09T17:51:38.816ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17322024-09-09T17:51:38.816ZINFOSledAgent (ServiceManager): Booting oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17332024-09-09T17:51:38.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 606.94306ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
17342024-09-09T17:51:38.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.053485744s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
17352024-09-09T17:51:38.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.297794818s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
17362024-09-09T17:51:38.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 817.408931ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
17372024-09-09T17:51:39.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.179422414s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
17382024-09-09T17:51:39.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 34.872816ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17392024-09-09T17:51:39.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 46.970258ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17402024-09-09T17:51:39.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 42.421901ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
17412024-09-09T17:51:39.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 75.229862ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17422024-09-09T17:51:39.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.316237026s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
17432024-09-09T17:51:39.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.496551104s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
17442024-09-09T17:51:39.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 123.916531ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17452024-09-09T17:51:39.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 148.126644ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
17462024-09-09T17:51:39.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.443943552s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
17472024-09-09T17:51:39.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.017134934s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
17482024-09-09T17:51:39.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 210.098978ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17492024-09-09T17:51:39.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 602.54197ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
17502024-09-09T17:51:39.308ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 296.740295ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17512024-09-09T17:51:39.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 125.928971ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
17522024-09-09T17:51:39.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.382709342s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
17532024-09-09T17:51:39.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 587.20827ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17542024-09-09T17:51:39.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.48015602s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
17552024-09-09T17:51:39.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.251640375s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
17562024-09-09T17:51:39.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 388.292463ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
17572024-09-09T17:51:39.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.049774216s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
17582024-09-09T17:51:39.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 287.086005ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17592024-09-09T17:51:39.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 877.667803ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
17602024-09-09T17:51:39.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 576.281966ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
17612024-09-09T17:51:39.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.031250043s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
17622024-09-09T17:51:39.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 542.090686ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17632024-09-09T17:51:39.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.286318902s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
17642024-09-09T17:51:40.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 458.131037ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17652024-09-09T17:51:40.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 878.276178ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
17662024-09-09T17:51:40.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 903.521922ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
17672024-09-09T17:51:40.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.022879195s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
17682024-09-09T17:51:40.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.457790947s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
17692024-09-09T17:51:40.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 986.331453ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
17702024-09-09T17:51:40.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 727.54632ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17712024-09-09T17:51:40.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 656.472438ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17722024-09-09T17:51:40.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 716.640775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
17732024-09-09T17:51:40.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.163457164s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
17742024-09-09T17:51:40.642ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17752024-09-09T17:51:40.643ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
17762024-09-09T17:51:40.643ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
17772024-09-09T17:51:40.649ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
17782024-09-09T17:51:40.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 562.543234ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
17792024-09-09T17:51:40.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.283659882s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
17802024-09-09T17:51:40.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 791.716135ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
17812024-09-09T17:51:40.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 699.203219ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
17822024-09-09T17:51:40.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 40.209819ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
17832024-09-09T17:51:40.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 33.599207ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
17842024-09-09T17:51:40.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 110.679051ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
17852024-09-09T17:51:40.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 124.684576ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
17862024-09-09T17:51:40.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.443618913s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
17872024-09-09T17:51:40.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 661.761608ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
17882024-09-09T17:51:41.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 249.446063ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
17892024-09-09T17:51:41.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 210.167964ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
17902024-09-09T17:51:41.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.345805026s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
17912024-09-09T17:51:41.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 796.106192ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
17922024-09-09T17:51:41.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.440535764s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
17932024-09-09T17:51:41.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 536.197439ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
17942024-09-09T17:51:41.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.201275474s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
17952024-09-09T17:51:41.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 432.328227ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
17962024-09-09T17:51:41.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 831.458021ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
17972024-09-09T17:51:41.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 245.29092ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
17982024-09-09T17:51:41.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.332576204s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
17992024-09-09T17:51:41.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.486865765s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
18002024-09-09T17:51:41.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.186413357s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
18012024-09-09T17:51:41.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.016247135s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
18022024-09-09T17:51:41.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.371992652s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
18032024-09-09T17:51:41.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.015984742s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
18042024-09-09T17:51:41.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.048938677s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
18052024-09-09T17:51:41.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 618.212961ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
18062024-09-09T17:51:41.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.116059926s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
18072024-09-09T17:51:41.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 554.931487ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
18082024-09-09T17:51:41.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.227634703s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
18092024-09-09T17:51:41.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.431698183s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
18102024-09-09T17:51:42.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.30753458s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
18112024-09-09T17:51:42.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.443536484s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
18122024-09-09T17:51:42.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 925.291351ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
18132024-09-09T17:51:42.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.308916056s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
18142024-09-09T17:51:42.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 720.368645ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
18152024-09-09T17:51:42.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.276008207s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
18162024-09-09T17:51:42.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 946.215415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
18172024-09-09T17:51:42.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.392406559s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
18182024-09-09T17:51:42.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.450554763s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
18192024-09-09T17:51:42.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.49063752s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
18202024-09-09T17:51:42.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.172163154s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
18212024-09-09T17:51:42.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 801.823076ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
18222024-09-09T17:51:42.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.431541523s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
18232024-09-09T17:51:42.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.370573243s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
18242024-09-09T17:51:42.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.123341314s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
18252024-09-09T17:51:42.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 968.67842ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
18262024-09-09T17:51:43.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.434830822s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
18272024-09-09T17:51:43.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.329886958s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
18282024-09-09T17:51:43.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.416197456s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
18292024-09-09T17:51:43.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 939.06358ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
18302024-09-09T17:51:43.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.14217941s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
18312024-09-09T17:51:43.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.297183669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
18322024-09-09T17:51:43.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 911.398111ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
18332024-09-09T17:51:43.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.022259566s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
18342024-09-09T17:51:43.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.07654003s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
18352024-09-09T17:51:43.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.460796279s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
18362024-09-09T17:51:43.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 575.011052ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
18372024-09-09T17:51:43.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 535.85692ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
18382024-09-09T17:51:43.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.481764425s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
18392024-09-09T17:51:44.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 667.567247ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
18402024-09-09T17:51:44.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.215558291s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
18412024-09-09T17:51:44.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.063270791s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
18422024-09-09T17:51:44.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 705.515404ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
18432024-09-09T17:51:44.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.239364001s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
18442024-09-09T17:51:44.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.095075654s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
18452024-09-09T17:51:44.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 550.245282ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
18462024-09-09T17:51:44.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 895.383023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
18472024-09-09T17:51:44.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 762.915905ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
18482024-09-09T17:51:44.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.427527426s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
18492024-09-09T17:51:44.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 673.29731ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
18502024-09-09T17:51:44.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 518.261258ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
18512024-09-09T17:51:44.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 632.151032ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
18522024-09-09T17:51:44.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.341715898s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
18532024-09-09T17:51:44.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.217893353s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
18542024-09-09T17:51:44.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 980.717863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
18552024-09-09T17:51:44.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.160612432s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
18562024-09-09T17:51:44.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 884.18341ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
18572024-09-09T17:51:44.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 570.026457ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
18582024-09-09T17:51:45.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 931.016988ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
18592024-09-09T17:51:45.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.224228462s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
18602024-09-09T17:51:45.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 916.479063ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
18612024-09-09T17:51:45.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 856.445655ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
18622024-09-09T17:51:45.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 803.025062ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
18632024-09-09T17:51:45.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 517.018915ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
18642024-09-09T17:51:45.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.296621843s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
18652024-09-09T17:51:45.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 797.387575ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
18662024-09-09T17:51:45.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.009450294s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
18672024-09-09T17:51:45.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.286869672s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
18682024-09-09T17:51:45.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 629.594566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
18692024-09-09T17:51:45.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.40725105s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
18702024-09-09T17:51:45.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 583.546116ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
18712024-09-09T17:51:45.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 842.720693ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
18722024-09-09T17:51:45.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.135541126s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
18732024-09-09T17:51:45.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.471634626s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
18742024-09-09T17:51:45.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 643.918449ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
18752024-09-09T17:51:45.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.12371746s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
18762024-09-09T17:51:46.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 652.218945ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
18772024-09-09T17:51:46.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.407014631s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
18782024-09-09T17:51:46.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.479511053s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
18792024-09-09T17:51:46.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.358877499s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
18802024-09-09T17:51:46.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.397206038s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
18812024-09-09T17:51:46.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 921.817704ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
18822024-09-09T17:51:46.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.381478552s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
18832024-09-09T17:51:46.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 878.816016ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
18842024-09-09T17:51:46.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.216853993s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
18852024-09-09T17:51:46.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 875.315783ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
18862024-09-09T17:51:46.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 988.925629ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
18872024-09-09T17:51:46.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 673.670842ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
18882024-09-09T17:51:46.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 818.448227ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
18892024-09-09T17:51:46.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.117961865s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
18902024-09-09T17:51:46.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.024774269s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
18912024-09-09T17:51:46.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.100995993s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
18922024-09-09T17:51:46.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.30567412s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
18932024-09-09T17:51:47.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.469464743s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
18942024-09-09T17:51:47.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.464571858s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
18952024-09-09T17:51:47.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.428259433s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
18962024-09-09T17:51:47.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.363782464s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
18972024-09-09T17:51:47.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 784.662364ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
18982024-09-09T17:51:47.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.245289802s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
18992024-09-09T17:51:47.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.000765116s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
19002024-09-09T17:51:47.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.45370398s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
19012024-09-09T17:51:47.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.183457627s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
19022024-09-09T17:51:47.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.065134349s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
19032024-09-09T17:51:47.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 589.829728ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
19042024-09-09T17:51:47.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 878.728984ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
19052024-09-09T17:51:47.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.000980568s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
19062024-09-09T17:51:47.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.144424713s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
19072024-09-09T17:51:47.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 983.045139ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
19082024-09-09T17:51:47.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 682.76403ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
19092024-09-09T17:51:48.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.453138492s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
19102024-09-09T17:51:48.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.376160661s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
19112024-09-09T17:51:48.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 970.517628ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
19122024-09-09T17:51:48.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 882.634632ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
19132024-09-09T17:51:48.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.426370815s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
19142024-09-09T17:51:48.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 571.000677ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
19152024-09-09T17:51:48.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.098055091s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
19162024-09-09T17:51:48.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 951.696308ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
19172024-09-09T17:51:48.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.388081749s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
19182024-09-09T17:51:48.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.141720208s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
19192024-09-09T17:51:48.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 651.927963ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
19202024-09-09T17:51:48.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.464809532s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
19212024-09-09T17:51:48.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.190873262s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
19222024-09-09T17:51:48.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.367547498s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
19232024-09-09T17:51:48.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.267397116s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
19242024-09-09T17:51:48.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 715.00987ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
19252024-09-09T17:51:48.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.025838525s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
19262024-09-09T17:51:48.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 774.006801ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
19272024-09-09T17:51:49.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.15862267s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
19282024-09-09T17:51:49.120ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="clickhouse_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19292024-09-09T17:51:49.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.178367503s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
19302024-09-09T17:51:49.177ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19312024-09-09T17:51:49.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.245638461s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
19322024-09-09T17:51:49.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 911.352538ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
19332024-09-09T17:51:49.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 36.889571ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19342024-09-09T17:51:49.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 123.164038ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19352024-09-09T17:51:49.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.418477919s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
19362024-09-09T17:51:49.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.377659201s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
19372024-09-09T17:51:49.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.285192642s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
19382024-09-09T17:51:49.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.124694517s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
19392024-09-09T17:51:49.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 242.968913ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19402024-09-09T17:51:49.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 730.862239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
19412024-09-09T17:51:49.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.16464199s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
19422024-09-09T17:51:49.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 753.267117ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
19432024-09-09T17:51:49.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 537.832171ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19442024-09-09T17:51:49.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 598.893114ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
19452024-09-09T17:51:49.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 664.531863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
19462024-09-09T17:51:49.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 527.836385ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
19472024-09-09T17:51:50.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 609.933448ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
19482024-09-09T17:51:50.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.150984508s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
19492024-09-09T17:51:50.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 583.349222ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
19502024-09-09T17:51:50.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.424971645s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
19512024-09-09T17:51:50.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 772.957299ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
19522024-09-09T17:51:50.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 683.05629ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
19532024-09-09T17:51:50.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 1.093114039s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19542024-09-09T17:51:50.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 740.496737ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
19552024-09-09T17:51:50.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 639.114076ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
19562024-09-09T17:51:50.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.139898547s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
19572024-09-09T17:51:50.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 896.371416ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
19582024-09-09T17:51:50.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 907.739865ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
19592024-09-09T17:51:50.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.070406174s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
19602024-09-09T17:51:50.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.170633837s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
19612024-09-09T17:51:50.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.458599562s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
19622024-09-09T17:51:50.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.409060052s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
19632024-09-09T17:51:50.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 871.54477ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
19642024-09-09T17:51:50.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 948.159712ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
19652024-09-09T17:51:50.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 757.561537ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
19662024-09-09T17:51:50.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 519.241469ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
19672024-09-09T17:51:50.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 840.884541ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
19682024-09-09T17:51:50.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 952.131112ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
19692024-09-09T17:51:50.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.177355221s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
19702024-09-09T17:51:51.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.376659797s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
19712024-09-09T17:51:51.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 946.596074ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
19722024-09-09T17:51:51.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 922.546942ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
19732024-09-09T17:51:51.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.183524246s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
19742024-09-09T17:51:51.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.284775607s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
19752024-09-09T17:51:51.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.213004552s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
19762024-09-09T17:51:51.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 624.775805ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19772024-09-09T17:51:51.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.163773874s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
19782024-09-09T17:51:51.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.470376152s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
19792024-09-09T17:51:51.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 806.65473ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
19802024-09-09T17:51:51.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.222099958s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
19812024-09-09T17:51:51.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 580.968171ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
19822024-09-09T17:51:51.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 899.416607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
19832024-09-09T17:51:51.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 593.326835ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
19842024-09-09T17:51:51.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 589.363785ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
19852024-09-09T17:51:51.905ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 13.321839016s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
19862024-09-09T17:51:51.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.223420416s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
19872024-09-09T17:51:52.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 813.01476ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
19882024-09-09T17:51:52.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 554.574619ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
19892024-09-09T17:51:52.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.234514351s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
19902024-09-09T17:51:52.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 985.91062ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
19912024-09-09T17:51:52.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 1.113019471s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
19922024-09-09T17:51:52.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 743.09984ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
19932024-09-09T17:51:52.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 771.652263ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
19942024-09-09T17:51:52.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 872.096749ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
19952024-09-09T17:51:52.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.229605257s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
19962024-09-09T17:51:52.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 750.543287ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
19972024-09-09T17:51:52.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.460547011s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
19982024-09-09T17:51:52.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 880.137535ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
19992024-09-09T17:51:52.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.111051692s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
20002024-09-09T17:51:52.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.243515848s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
20012024-09-09T17:51:52.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.250858133s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
20022024-09-09T17:51:52.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.016092139s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
20032024-09-09T17:51:52.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.466424884s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
20042024-09-09T17:51:52.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 572.412174ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
20052024-09-09T17:51:52.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 934.239321ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
20062024-09-09T17:51:52.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 715.21702ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
20072024-09-09T17:51:53.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.003490458s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
20082024-09-09T17:51:53.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 972.715643ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
20092024-09-09T17:51:53.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.482380221s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
20102024-09-09T17:51:53.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 942.843885ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
20112024-09-09T17:51:53.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.29610942s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
20122024-09-09T17:51:53.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 1.452311086s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
20132024-09-09T17:51:53.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.464348256s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
20142024-09-09T17:51:53.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 699.046176ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
20152024-09-09T17:51:53.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.457635322s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
20162024-09-09T17:51:53.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 833.264699ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
20172024-09-09T17:51:53.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 525.573406ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
20182024-09-09T17:51:53.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.381171288s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
20192024-09-09T17:51:53.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.139514104s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
20202024-09-09T17:51:53.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 837.778632ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
20212024-09-09T17:51:53.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.488021547s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
20222024-09-09T17:51:53.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 769.777378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
20232024-09-09T17:51:53.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 969.833856ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
20242024-09-09T17:51:53.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.243944904s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
20252024-09-09T17:51:54.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 808.268956ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
20262024-09-09T17:51:54.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 732.287885ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
20272024-09-09T17:51:54.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.048967252s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
20282024-09-09T17:51:54.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 828.205503ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
20292024-09-09T17:51:54.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 590.690838ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
20302024-09-09T17:51:54.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 674.301099ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
20312024-09-09T17:51:54.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.202307373s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
20322024-09-09T17:51:54.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 571.79789ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
20332024-09-09T17:51:54.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 686.87844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
20342024-09-09T17:51:54.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.38753995s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
20352024-09-09T17:51:54.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 751.732027ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
20362024-09-09T17:51:54.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 1.008349264s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
20372024-09-09T17:51:54.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 806.602026ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
20382024-09-09T17:51:54.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 951.177319ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
20392024-09-09T17:51:54.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.077116966s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
20402024-09-09T17:51:54.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.469371636s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
20412024-09-09T17:51:54.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.276979825s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
20422024-09-09T17:51:54.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 660.94128ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
20432024-09-09T17:51:54.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 869.260452ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
20442024-09-09T17:51:54.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.49427488s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
20452024-09-09T17:51:54.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.195397363s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
20462024-09-09T17:51:55.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 847.936825ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
20472024-09-09T17:51:55.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 922.171435ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
20482024-09-09T17:51:55.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544") failed: Property not found. retry in 1.493448445s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
20492024-09-09T17:51:55.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.13184171s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
20502024-09-09T17:51:55.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 517.880104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
20512024-09-09T17:51:55.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.000720785s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
20522024-09-09T17:51:55.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393") failed: Property not found. retry in 1.280797197s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
20532024-09-09T17:51:55.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 906.820533ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
20542024-09-09T17:51:55.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.284302789s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
20552024-09-09T17:51:55.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.051981989s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
20562024-09-09T17:51:55.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 919.903141ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
20572024-09-09T17:51:55.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 563.203542ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
20582024-09-09T17:51:55.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.205955988s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
20592024-09-09T17:51:55.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.05826215s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
20602024-09-09T17:51:55.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 669.335813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
20612024-09-09T17:51:55.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.000134546s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
20622024-09-09T17:51:56.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 881.701726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
20632024-09-09T17:51:56.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 514.605145ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
20642024-09-09T17:51:56.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 1.166160504s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
20652024-09-09T17:51:56.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.148657641s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
20662024-09-09T17:51:56.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 560.944783ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
20672024-09-09T17:51:56.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 1.333663281s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
20682024-09-09T17:51:56.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.249871302s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
20692024-09-09T17:51:56.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.047102597s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
20702024-09-09T17:51:56.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 1.262473247s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
20712024-09-09T17:51:56.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 876.173275ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
20722024-09-09T17:51:56.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 850.31421ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
20732024-09-09T17:51:56.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.038432955s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
20742024-09-09T17:51:56.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 860.402308ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
20752024-09-09T17:51:56.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.054114271s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
20762024-09-09T17:51:56.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639") failed: Property not found. retry in 1.432183553s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
20772024-09-09T17:51:56.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 1.006497335s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
20782024-09-09T17:51:56.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.438743674s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
20792024-09-09T17:51:56.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 673.298657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
20802024-09-09T17:51:56.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 1.185629549s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
20812024-09-09T17:51:57.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.086963538s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
20822024-09-09T17:51:57.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de") failed: Property not found. retry in 1.422306737s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
20832024-09-09T17:51:57.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8") failed: Property not found. retry in 1.423786761s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
20842024-09-09T17:51:57.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 968.116691ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
20852024-09-09T17:51:57.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.122135877s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
20862024-09-09T17:51:57.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 661.349001ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
20872024-09-09T17:51:57.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 1.168187917s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
20882024-09-09T17:51:57.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 535.52757ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
20892024-09-09T17:51:57.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb") failed: Property not found. retry in 1.030856842s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
20902024-09-09T17:51:57.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03") failed: Property not found. retry in 1.369058632s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
20912024-09-09T17:51:57.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e") failed: Property not found. retry in 610.350581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
20922024-09-09T17:51:57.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f") failed: Property not found. retry in 511.331268ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
20932024-09-09T17:51:57.726ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38191
20942024-09-09T17:51:57.729ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1329 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38191 req_id = dd86d688-4c95-45b4-88f2-fff9b602f06b response_code = 200 uri = /baseboard
20952024-09-09T17:51:57.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 1.314391934s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
20962024-09-09T17:51:57.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 1.218069302s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
20972024-09-09T17:51:58.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07") failed: Property not found. retry in 1.08508415s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
20982024-09-09T17:51:58.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 897.493148ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
20992024-09-09T17:51:58.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9") failed: Property not found. retry in 631.633742ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
21002024-09-09T17:51:58.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06") failed: Property not found. retry in 1.436862085s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
21012024-09-09T17:51:58.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 504.369415ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
21022024-09-09T17:51:58.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148") failed: Property not found. retry in 1.42019473s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
21032024-09-09T17:51:58.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065") failed: Property not found. retry in 1.026979257s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
21042024-09-09T17:51:58.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda") failed: Property not found. retry in 902.624786ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
21052024-09-09T17:51:58.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3") failed: Property not found. retry in 675.253077ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
21062024-09-09T17:51:59.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823") failed: Property not found. retry in 1.366039973s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
21072024-09-09T17:51:59.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d") failed: Property not found. retry in 787.683123ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
21082024-09-09T17:51:59.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc") failed: Property not found. retry in 903.778023ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
21092024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_external_dns_07c1bbdd-f080-4837-9528-41234be47bda
21102024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_899a683c-bd60-4b15-b12a-e5db4452f148
21112024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_nexus_fd75f244-5e91-4f63-b3b2-ae4139e91823
21122024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_oximeter_fe9e311d-41c3-4a6e-8e3e-3fbb54dee639
21132024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_c52a5621-100d-4082-b483-9a9b71016544
21142024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_nexus_ca022d43-3da4-4507-941c-6fd9da5b668d
21152024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_e37b75a6-6c18-4702-a94d-ecb36cc45c07
21162024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_external_dns_9711344c-11c4-490b-9560-e9c007eb0cf3
21172024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_clickhouse_e017ec87-832c-4092-8516-57b23da956bc
21182024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_18d54208-fbd9-4a9a-bb4c-0dda56eeac06
21192024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_0d731515-3696-45fe-816a-2c2bf0393b4f
21202024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_nexus_4c6ef9f1-4001-4c65-ae99-bafe915b66f8
21212024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_pantry_4a391a53-62d4-4c58-a8df-837ca0cab6d9
21222024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_a4f04eb1-0d05-41b6-b3d2-1d063e4e900e
21232024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_pantry_bebbab47-f19c-4831-a25e-344318d56393
21242024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_88cec213-8d41-4e46-aea3-2d037677a065
21252024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_pantry_ef3a2fd8-3786-4198-8c2d-a9442073f2de
21262024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_2ce49a1d-0f2c-40bf-9a41-d49a169c3abb
21272024-09-09T17:52:00.486ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_a6fd4f40-6a6d-4e01-a5a7-0e34b5f5ae03
21282024-09-09T17:52:00.489ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
21292024-09-09T17:52:00.496ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
21302024-09-09T17:52:00.505ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 33091398 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59416 req_id = 9a469a76-416e-4383-aa32-659685becd61 response_code = 204 uri = /omicron-zones
21312024-09-09T17:52:00.518ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1286
21322024-09-09T17:52:00.518ZINFOSledAgent (RSS): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
21332024-09-09T17:52:00.518ZINFOSledAgent (RSS): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
21342024-09-09T17:52:00.525ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:687
21352024-09-09T17:52:00.528ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:700
21362024-09-09T17:52:00.594ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:867
21372024-09-09T17:52:00.613ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21382024-09-09T17:52:00.668ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21392024-09-09T17:52:00.833ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21402024-09-09T17:52:00.945ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21412024-09-09T17:52:01.179ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21422024-09-09T17:52:01.369ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21432024-09-09T17:52:01.657ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21442024-09-09T17:52:01.913ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21452024-09-09T17:52:02.252ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21462024-09-09T17:52:02.497ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21472024-09-09T17:52:02.763ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21482024-09-09T17:52:03.461ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21492024-09-09T17:52:04.172ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21502024-09-09T17:52:05.265ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 26.381685121s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
21512024-09-09T17:52:05.451ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21522024-09-09T17:52:06.965ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21532024-09-09T17:52:07.792ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21542024-09-09T17:52:09.319ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21552024-09-09T17:52:10.819ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21562024-09-09T17:52:12.627ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21572024-09-09T17:52:16.497ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21582024-09-09T17:52:19.392ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/1068bc7e-e3ab-44f6-b097-299854b0081f/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
21592024-09-09T17:52:22.265ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "7bf4982b-676a-4f39-a13a-42c9f45b95d7", "content-length": "173", "date": "Mon, 09 Sep 2024 17:52:22 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"1068bc7e-e3ab-44f6-b097-299854b0081f\"", request_id: "7bf4982b-676a-4f39-a13a-42c9f45b95d7" } file = sled-agent/src/rack_setup/service.rs:929
21602024-09-09T17:52:27.766ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64845
21612024-09-09T17:52:27.767ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 620 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64845 req_id = c5887270-4a30-41af-ac42-9d446360eff8 response_code = 200 uri = /baseboard
21622024-09-09T17:52:29.675ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "c5d09c0d-7435-4394-b97e-9d4c49d8ef7e", "content-length": "173", "date": "Mon, 09 Sep 2024 17:52:29 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"61e07fcc-45b4-4b25-ae0d-e78eee62eb60\"", request_id: "c5d09c0d-7435-4394-b97e-9d4c49d8ef7e" } file = sled-agent/src/rack_setup/service.rs:929
21632024-09-09T17:52:34.125ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "3762bf19-6e73-4127-ac27-2075056b81e9", "content-length": "173", "date": "Mon, 09 Sep 2024 17:52:34 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"61e07fcc-45b4-4b25-ae0d-e78eee62eb60\"", request_id: "3762bf19-6e73-4127-ac27-2075056b81e9" } file = sled-agent/src/rack_setup/service.rs:929
21642024-09-09T17:52:44.255ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "ce3454ed-b17f-4516-9246-53b32f21fc3a", "content-length": "173", "date": "Mon, 09 Sep 2024 17:52:44 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"61e07fcc-45b4-4b25-ae0d-e78eee62eb60\"", request_id: "ce3454ed-b17f-4516-9246-53b32f21fc3a" } file = sled-agent/src/rack_setup/service.rs:929
21652024-09-09T17:52:49.524ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
21662024-09-09T17:52:49.613ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40202
21672024-09-09T17:52:49.617ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21682024-09-09T17:52:49.706ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92452 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40202 req_id = b8206ce8-5f79-45dc-a84e-1b19b4de4809 response_code = 200 uri = /inventory
21692024-09-09T17:52:49.709ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21702024-09-09T17:52:49.709ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
21712024-09-09T17:52:49.720ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
21722024-09-09T17:52:49.721ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 13633 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40202 req_id = 90c5717e-7e3f-40a9-88d8-55fc8072d767 response_code = 200 uri = /omicron-zones
21732024-09-09T17:52:53.607ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54498
21742024-09-09T17:52:53.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2577 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54498 req_id = a809104e-4bea-4165-b23a-dce4633a0251 response_code = 200 uri = /vpc-routes
21752024-09-09T17:52:53.613ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37747
21762024-09-09T17:52:53.613ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37747 req_id = dd35b4b7-76cc-489f-b6c2-120802ad3eec response_code = 200 uri = /vpc-routes
21772024-09-09T17:52:53.659ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49696
21782024-09-09T17:52:53.663ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4588 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49696 req_id = b165ecd7-9979-4e52-8037-e367c5aabc72 response_code = 200 uri = /v2p
21792024-09-09T17:52:53.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 12825 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37747 req_id = 0adc7c06-5137-493e-8b76-45de23ec0236 response_code = 204 uri = /vpc-routes
21802024-09-09T17:52:53.742ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42419
21812024-09-09T17:52:53.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42419 req_id = 9623d4ac-a627-43d8-87a8-af2425da9763 response_code = 200 uri = /v2p
21822024-09-09T17:52:53.753ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54498 req_id = bdbe4b58-a7ae-420d-8ed1-583cc663e864 response_code = 204 uri = /vpc-routes
21832024-09-09T17:52:53.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52440
21842024-09-09T17:52:53.940ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52440 req_id = bd3d35f8-1fd5-4546-bfdb-999731fdbaf3 response_code = 200 uri = /v2p
21852024-09-09T17:52:53.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63440
21862024-09-09T17:52:53.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63440 req_id = fda4aeca-1197-4909-90c3-b6dd15d78316 response_code = 200 uri = /vpc-routes
21872024-09-09T17:52:53.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63440 req_id = 157b50c4-cd30-4c06-92b1-8d1bf7a70f3e response_code = 204 uri = /vpc-routes
21882024-09-09T17:52:54.789ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64563
21892024-09-09T17:52:54.794ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
21902024-09-09T17:52:54.794ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
21912024-09-09T17:52:54.794ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21922024-09-09T17:52:54.794ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21932024-09-09T17:52:54.794ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21942024-09-09T17:52:54.794ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21952024-09-09T17:52:54.794ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21962024-09-09T17:52:54.794ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5667 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64563 req_id = 521d2f0e-6512-419f-bea2-29a91399f4c3 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
21972024-09-09T17:52:54.795ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:939
21982024-09-09T17:52:54.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45364
21992024-09-09T17:52:54.990ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22002024-09-09T17:52:55.079ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89077 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45364 req_id = 14d7aed0-1daa-40cf-a2ce-72fd2c5532e6 response_code = 200 uri = /inventory
22012024-09-09T17:52:55.081ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22022024-09-09T17:52:55.081ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/config/omicron-zones.json file = common/src/ledger.rs:177
22032024-09-09T17:52:55.081ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/config/omicron-zones.json file = common/src/ledger.rs:177
22042024-09-09T17:52:55.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 995 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45364 req_id = 41aac253-afae-4548-9506-50648ebec531 response_code = 200 uri = /omicron-zones
22052024-09-09T17:52:55.377ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39528
22062024-09-09T17:52:55.377ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22072024-09-09T17:52:55.377ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
22082024-09-09T17:52:55.377ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22092024-09-09T17:52:55.377ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22102024-09-09T17:52:55.377ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22112024-09-09T17:52:55.377ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22122024-09-09T17:52:55.378ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22132024-09-09T17:52:55.378ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 377 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39528 req_id = bd402fbd-c9ac-43a6-b03e-e1c21c4a8cdc response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22142024-09-09T17:52:55.389ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44802
22152024-09-09T17:52:55.389ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22162024-09-09T17:52:55.389ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
22172024-09-09T17:52:55.389ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22182024-09-09T17:52:55.389ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22192024-09-09T17:52:55.389ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22202024-09-09T17:52:55.389ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22212024-09-09T17:52:55.389ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22222024-09-09T17:52:55.389ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44802 req_id = 18dac836-493f-402c-a8e9-60b9b5795154 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22232024-09-09T17:52:55.657ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56987
22242024-09-09T17:52:55.658ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22252024-09-09T17:52:55.658ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
22262024-09-09T17:52:55.658ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22272024-09-09T17:52:55.658ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22282024-09-09T17:52:55.658ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22292024-09-09T17:52:55.658ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22302024-09-09T17:52:55.658ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22312024-09-09T17:52:55.658ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 311 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56987 req_id = 04a521ef-bac9-40ad-9039-25a92712c39d response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22322024-09-09T17:52:57.802ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41797
22332024-09-09T17:52:57.803ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41797 req_id = 5b8e9468-9bed-4a76-af3b-be71519755dc response_code = 200 uri = /baseboard
22342024-09-09T17:52:59.778ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43106
22352024-09-09T17:52:59.778ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43106 req_id = fa57e783-cd69-434a-993e-4ac66f7bcb56 response_code = 200 uri = /vpc-routes
22362024-09-09T17:52:59.788ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43106 req_id = f9ce0af2-8936-4e67-ad20-515332f841c6 response_code = 204 uri = /vpc-routes
22372024-09-09T17:53:01.789ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:42327 remote_addr = [fd00:1122:3344:101::d]:47540 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60
22382024-09-09T17:53:01.831ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39496 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = a081cdd6-253e-4509-8760-337136bd44d9 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
22392024-09-09T17:53:23.728ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39803
22402024-09-09T17:53:23.728ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39803 req_id = e4ff4ea1-25be-4916-8634-9521735d3984 response_code = 200 uri = /vpc-routes
22412024-09-09T17:53:23.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:39803 req_id = 1113c43d-a049-4b50-a6a7-2f6007fd6ec3 response_code = 204 uri = /vpc-routes
22422024-09-09T17:53:23.799ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48346
22432024-09-09T17:53:23.799ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 222 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48346 req_id = 372aa1bf-423a-44f3-a7c4-5e73a54ccd24 response_code = 200 uri = /v2p
22442024-09-09T17:53:23.802ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47389
22452024-09-09T17:53:23.802ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47389 req_id = 5b50f4e7-b05b-4c1a-b5cc-4aa369fe6590 response_code = 200 uri = /vpc-routes
22462024-09-09T17:53:23.822ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1761 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47389 req_id = 5b48e1e3-d04b-49e1-b882-98aefd653cb5 response_code = 204 uri = /vpc-routes
22472024-09-09T17:53:23.955ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63413
22482024-09-09T17:53:23.955ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63413 req_id = 5de360cf-9aba-4676-82e2-3669c49d4430 response_code = 200 uri = /v2p
22492024-09-09T17:53:24.059ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49462
22502024-09-09T17:53:24.063ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
22512024-09-09T17:53:24.138ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
22522024-09-09T17:53:24.204ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144748 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49462 req_id = 5437fdfb-0755-41c9-a4b7-2d0055a31502 response_code = 204 uri = /switch-ports
22532024-09-09T17:53:24.204ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56193
22542024-09-09T17:53:24.204ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37158
22552024-09-09T17:53:24.204ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46988
22562024-09-09T17:53:24.205ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 949 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56193 req_id = 9c41ef6c-f605-4e33-a18b-0bfa04664b2d response_code = 200 uri = /vpc-routes
22572024-09-09T17:53:24.205ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
22582024-09-09T17:53:24.209ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4523 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37158 req_id = 203a2d6d-5c33-4801-9ec1-34a2b699cfeb response_code = 200 uri = /v2p
22592024-09-09T17:53:24.215ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 10680 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49462 req_id = 93ff8dc5-a216-4d9a-92af-9b8608be69ed response_code = 200 uri = /network-bootstore-config
22602024-09-09T17:53:24.216ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 778 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56193 req_id = 5b268211-be3a-4c24-be55-6d5ccb9301ae response_code = 204 uri = /vpc-routes
22612024-09-09T17:53:24.251ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
22622024-09-09T17:53:24.268ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
22632024-09-09T17:53:24.269ZINFOSledAgent (bootstore): Writing ledger to /pool/int/d05e61ff-a727-4851-80ca-eab919525a00/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
22642024-09-09T17:53:24.270ZINFOSledAgent (bootstore): Writing ledger to /pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
22652024-09-09T17:53:24.271ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
22662024-09-09T17:53:24.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 11415 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49462 req_id = 9efd80c9-b54b-41e5-bde7-ece5a525018a response_code = 204 uri = /network-bootstore-config
22672024-09-09T17:53:24.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108631 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46988 req_id = 64fd0c68-c613-4025-8352-e4f0b1f7a637 response_code = 204 uri = /switch-ports
22682024-09-09T17:53:24.314ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 408 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46988 req_id = cdd9b6c9-f2e8-4347-88b3-298e0ec8d5be response_code = 200 uri = /network-bootstore-config
22692024-09-09T17:53:24.360ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64715
22702024-09-09T17:53:24.360ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
22712024-09-09T17:53:24.397ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
22722024-09-09T17:53:24.457ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97335 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:64715 req_id = 60c463a0-346e-401e-9572-f1b3f3722f50 response_code = 204 uri = /switch-ports
22732024-09-09T17:53:24.458ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 664 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64715 req_id = c6961f5b-4110-48a0-9381-ad6e35b2cfa3 response_code = 200 uri = /network-bootstore-config
22742024-09-09T17:53:27.836ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40161
22752024-09-09T17:53:27.836ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40161 req_id = 37e58b2f-240d-449c-ae46-a259cb04c19f response_code = 200 uri = /baseboard
22762024-09-09T17:53:31.794ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7050 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = e55ea249-124a-4357-95f6-86e21aee611e response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
22772024-09-09T17:53:53.588ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57817
22782024-09-09T17:53:53.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57817 req_id = a4712d80-772a-4da2-8d4d-67533952b802 response_code = 200 uri = /vpc-routes
22792024-09-09T17:53:53.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1427 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57817 req_id = 06dfc1cb-001d-4896-9331-9fc785354d8b response_code = 204 uri = /vpc-routes
22802024-09-09T17:53:53.681ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41737
22812024-09-09T17:53:53.681ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41737 req_id = b4ff9b69-9c25-4264-bfdf-60c4fe86869d response_code = 200 uri = /vpc-routes
22822024-09-09T17:53:53.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41737 req_id = 7ce7c922-950f-4568-816e-09a73fba22f9 response_code = 204 uri = /vpc-routes
22832024-09-09T17:53:53.734ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63112
22842024-09-09T17:53:53.735ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63112 req_id = 240414b8-45cb-4520-868e-251f673de6fc response_code = 200 uri = /v2p
22852024-09-09T17:53:53.757ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50664
22862024-09-09T17:53:53.757ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50664 req_id = dc16c25b-0978-41d2-95f5-2f0dddcb1abc response_code = 200 uri = /v2p
22872024-09-09T17:53:53.799ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34299
22882024-09-09T17:53:53.800ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
22892024-09-09T17:53:53.851ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
22902024-09-09T17:53:53.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 122074 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34299 req_id = 1bd4e904-0d28-40aa-88e7-7331788a3ad3 response_code = 204 uri = /switch-ports
22912024-09-09T17:53:53.921ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48538
22922024-09-09T17:53:53.921ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
22932024-09-09T17:53:53.964ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
22942024-09-09T17:53:54.036ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114696 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48538 req_id = 8c216c13-8442-4562-9327-9a349f034d00 response_code = 204 uri = /switch-ports
22952024-09-09T17:53:54.036ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53151
22962024-09-09T17:53:54.036ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53151 req_id = f09cb747-7e8b-4e9e-a5e4-22e6c90c93de response_code = 200 uri = /vpc-routes
22972024-09-09T17:53:54.036ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 244 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34299 req_id = 7d63bb87-b4b8-404a-a76a-5653a54c5e8c response_code = 200 uri = /network-bootstore-config
22982024-09-09T17:53:54.036ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 255 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48538 req_id = a024a6a3-4b61-4f03-b954-cea9f15d3e28 response_code = 200 uri = /network-bootstore-config
22992024-09-09T17:53:54.047ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53151 req_id = 6c2dacce-d057-406d-8f11-035f38810a34 response_code = 204 uri = /vpc-routes
23002024-09-09T17:53:54.119ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45224
23012024-09-09T17:53:54.120ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 163 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45224 req_id = 83045ef0-ed3a-4e23-a75d-5913fd1306f0 response_code = 200 uri = /v2p
23022024-09-09T17:53:54.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54496
23032024-09-09T17:53:54.148ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23042024-09-09T17:53:54.191ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23052024-09-09T17:53:54.247ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98159 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54496 req_id = 7010bc56-6ab8-4bc1-a072-259e07964ead response_code = 204 uri = /switch-ports
23062024-09-09T17:53:54.247ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 695 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54496 req_id = 7b5a97b4-0863-47e0-8bd3-416565dca14b response_code = 200 uri = /network-bootstore-config
23072024-09-09T17:53:57.869ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40288
23082024-09-09T17:53:57.869ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 50 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40288 req_id = 43e05087-fd24-4347-bd73-ec4fe6e3a63b response_code = 200 uri = /baseboard
23092024-09-09T17:54:01.794ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6597 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 5a05a5d0-7417-4a9c-b938-2a958f7373d0 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
23102024-09-09T17:54:22.555ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
23112024-09-09T17:54:23.719ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47267
23122024-09-09T17:54:23.720ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1231 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47267 req_id = 1669fc29-c413-4b62-bcde-924ed44e5ebc response_code = 200 uri = /v2p
23132024-09-09T17:54:23.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44779
23142024-09-09T17:54:23.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44779 req_id = e1c7c8d6-2291-4dbf-b8b0-a31140b8a253 response_code = 200 uri = /vpc-routes
23152024-09-09T17:54:23.858ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44779 req_id = b173853e-831b-4403-979b-f99f41173232 response_code = 204 uri = /vpc-routes
23162024-09-09T17:54:23.876ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54221
23172024-09-09T17:54:23.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54221 req_id = da2f1713-cef7-495c-bf25-1ebed9da8b2d response_code = 200 uri = /vpc-routes
23182024-09-09T17:54:23.956ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42970
23192024-09-09T17:54:23.956ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42970 req_id = 87dd53cb-667b-4a10-b982-3ad37260196b response_code = 200 uri = /v2p
23202024-09-09T17:54:23.963ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54221 req_id = 64471f78-d4cc-4010-84df-0853fc9b6d93 response_code = 204 uri = /vpc-routes
23212024-09-09T17:54:23.986ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63699
23222024-09-09T17:54:23.986ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23232024-09-09T17:54:24.029ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23242024-09-09T17:54:24.104ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118623 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63699 req_id = f3f1ad21-77ae-4aa5-bb94-b37c3172b7bc response_code = 204 uri = /switch-ports
23252024-09-09T17:54:24.105ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63121
23262024-09-09T17:54:24.105ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64912
23272024-09-09T17:54:24.105ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 224 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63699 req_id = 245bb3eb-fa2f-4068-955c-7b47c0088570 response_code = 200 uri = /network-bootstore-config
23282024-09-09T17:54:24.105ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63121 req_id = f9465567-8898-4416-bac2-6874ec7c35a5 response_code = 200 uri = /vpc-routes
23292024-09-09T17:54:24.105ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23302024-09-09T17:54:24.148ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23312024-09-09T17:54:24.182ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47340
23322024-09-09T17:54:24.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 640 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47340 req_id = 4bdd6742-5e66-4257-b423-01304261aca7 response_code = 200 uri = /v2p
23332024-09-09T17:54:24.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 564 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63121 req_id = 5ba31cf1-be6b-4423-a474-46d4428e7f17 response_code = 204 uri = /vpc-routes
23342024-09-09T17:54:24.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101049 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64912 req_id = a6fc0a77-bc38-4a76-a85d-c5f4e1e95bde response_code = 204 uri = /switch-ports
23352024-09-09T17:54:24.207ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64912 req_id = 5cf7fa60-fbde-4df6-b93f-cd8bc52c37d7 response_code = 200 uri = /network-bootstore-config
23362024-09-09T17:54:24.219ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52715
23372024-09-09T17:54:24.220ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23382024-09-09T17:54:24.261ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23392024-09-09T17:54:24.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99066 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52715 req_id = 66a4f8e4-5d66-463c-9163-658255c00ec2 response_code = 204 uri = /switch-ports
23402024-09-09T17:54:24.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52715 req_id = a2b44de9-0bab-4737-a672-0b3d52f533a1 response_code = 200 uri = /network-bootstore-config
23412024-09-09T17:54:27.902ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61305
23422024-09-09T17:54:27.902ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61305 req_id = 24e1433a-412f-4e02-bf84-2f7432f9913c response_code = 200 uri = /baseboard
23432024-09-09T17:54:31.795ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6561 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 5dc7821f-0750-4e4d-aca7-d9840655dd65 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
23442024-09-09T17:54:53.636ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62014
23452024-09-09T17:54:53.636ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62014 req_id = 2e129638-2333-4696-8703-168863e90aea response_code = 200 uri = /v2p
23462024-09-09T17:54:53.646ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37779
23472024-09-09T17:54:53.646ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37779 req_id = 5c08ccfd-8a69-47ab-84e1-9f9f45ef9efb response_code = 200 uri = /vpc-routes
23482024-09-09T17:54:53.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37779 req_id = adb6fea2-764c-4adb-abf9-82ecd1c369c4 response_code = 204 uri = /vpc-routes
23492024-09-09T17:54:53.655ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60013
23502024-09-09T17:54:53.656ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 886 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60013 req_id = 1a27fe7d-c7c7-41f2-b240-5e63f37e0c39 response_code = 200 uri = /vpc-routes
23512024-09-09T17:54:53.667ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60013 req_id = 846f162d-27f2-43df-915a-861efcd9f639 response_code = 204 uri = /vpc-routes
23522024-09-09T17:54:53.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63096
23532024-09-09T17:54:53.747ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23542024-09-09T17:54:53.793ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23552024-09-09T17:54:53.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124390 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63096 req_id = 8cbbe9c6-cb08-4df1-a700-0ea7d296e510 response_code = 204 uri = /switch-ports
23562024-09-09T17:54:53.872ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63096 req_id = 191534b1-d67f-4a7b-b315-bb6e532318f2 response_code = 200 uri = /network-bootstore-config
23572024-09-09T17:54:53.901ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46570
23582024-09-09T17:54:53.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46570 req_id = 4e53ec1b-16dc-4c5d-be24-bd76fa2fc975 response_code = 200 uri = /v2p
23592024-09-09T17:54:53.931ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55199
23602024-09-09T17:54:53.931ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23612024-09-09T17:54:53.969ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23622024-09-09T17:54:54.027ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96002 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:55199 req_id = 35a91d40-00da-4603-82f1-c9f136e3d767 response_code = 204 uri = /switch-ports
23632024-09-09T17:54:54.027ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52066
23642024-09-09T17:54:54.027ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62835
23652024-09-09T17:54:54.027ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62835 req_id = f727efb4-c292-40e3-b8cc-87850baecfa4 response_code = 200 uri = /vpc-routes
23662024-09-09T17:54:54.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 292 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52066 req_id = 2d8b5dfc-cbce-429b-91bf-601e9cf3203e response_code = 200 uri = /v2p
23672024-09-09T17:54:54.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55199 req_id = 9cf5ed0e-f9e3-4cc7-ac95-a79be3bb1b3b response_code = 200 uri = /network-bootstore-config
23682024-09-09T17:54:54.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62835 req_id = fe22a831-0ba1-4cbf-9196-30c32c0a4102 response_code = 204 uri = /vpc-routes
23692024-09-09T17:54:54.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57231
23702024-09-09T17:54:54.047ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23712024-09-09T17:54:54.083ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23722024-09-09T17:54:54.136ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89428 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57231 req_id = ced738ca-5488-4d5e-9dcc-2fb8b9ad5c99 response_code = 204 uri = /switch-ports
23732024-09-09T17:54:54.137ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57231 req_id = 226a84a9-e384-4fae-9f95-c2de9078cc2f response_code = 200 uri = /network-bootstore-config
23742024-09-09T17:54:57.937ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36152
23752024-09-09T17:54:57.937ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 50 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36152 req_id = 9130651d-8d01-428e-a7a7-6d492badc589 response_code = 200 uri = /baseboard
23762024-09-09T17:55:01.797ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7868 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = c218c44e-ba44-4406-8c4e-936237ed86bd response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
23772024-09-09T17:55:23.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43294
23782024-09-09T17:55:23.661ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43294 req_id = c75e713c-9010-4ab2-9fff-cbed21c34ab9 response_code = 200 uri = /vpc-routes
23792024-09-09T17:55:23.683ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1414 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43294 req_id = 5ce676d3-9481-4654-b2c6-9b61ed6dc6d6 response_code = 204 uri = /vpc-routes
23802024-09-09T17:55:23.756ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52949
23812024-09-09T17:55:23.756ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52949 req_id = 7461d1bc-d2f0-4fc0-bf16-90eb7ed9170d response_code = 200 uri = /vpc-routes
23822024-09-09T17:55:23.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1429 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52949 req_id = 21637a23-8dda-4e0c-94fc-7fbbe508a31e response_code = 204 uri = /vpc-routes
23832024-09-09T17:55:23.903ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54503
23842024-09-09T17:55:23.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54503 req_id = 7e0d7d10-b115-4c52-a09c-3f8c3fba18a0 response_code = 200 uri = /v2p
23852024-09-09T17:55:23.914ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40904
23862024-09-09T17:55:23.915ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40904 req_id = 35728636-9548-43d3-856d-48b73b3d0895 response_code = 200 uri = /v2p
23872024-09-09T17:55:23.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43296
23882024-09-09T17:55:23.941ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23892024-09-09T17:55:23.983ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23902024-09-09T17:55:24.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103158 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43296 req_id = 3a72e51b-a23e-45c8-9a97-29b3817cf100 response_code = 204 uri = /switch-ports
23912024-09-09T17:55:24.044ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39800
23922024-09-09T17:55:24.044ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
23932024-09-09T17:55:24.076ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
23942024-09-09T17:55:24.132ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88002 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39800 req_id = db4cd447-c646-4440-8fe6-989796eaabb9 response_code = 204 uri = /switch-ports
23952024-09-09T17:55:24.132ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52925
23962024-09-09T17:55:24.132ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 284 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43296 req_id = f9a56da7-aad0-4b13-8af9-900e3c347fee response_code = 200 uri = /network-bootstore-config
23972024-09-09T17:55:24.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 968 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52925 req_id = 435066ae-4ffa-4ace-be80-c721cb8b5dfe response_code = 200 uri = /vpc-routes
23982024-09-09T17:55:24.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39800 req_id = 64d20417-2793-42e4-81c1-d422ff118722 response_code = 200 uri = /network-bootstore-config
23992024-09-09T17:55:24.241ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56467
24002024-09-09T17:55:24.242ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56467 req_id = 00a59e98-9389-42f1-aa62-ae01f4842da5 response_code = 200 uri = /v2p
24012024-09-09T17:55:24.252ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52925 req_id = e4ba353f-f38f-42f3-9648-7968a9c6e5aa response_code = 204 uri = /vpc-routes
24022024-09-09T17:55:24.330ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46446
24032024-09-09T17:55:24.330ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24042024-09-09T17:55:24.366ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24052024-09-09T17:55:24.423ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93356 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46446 req_id = 0d8ce8c0-0540-432d-805f-982a925b6864 response_code = 204 uri = /switch-ports
24062024-09-09T17:55:24.423ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46446 req_id = d06c2a23-caff-4ba2-8b3b-d275b7dc173e response_code = 200 uri = /network-bootstore-config
24072024-09-09T17:55:27.971ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58304
24082024-09-09T17:55:27.971ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 52 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58304 req_id = 4aea744e-b343-4940-a122-8e322ad32083 response_code = 200 uri = /baseboard
24092024-09-09T17:55:31.887ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8385 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 6ad06fed-9c20-4642-b688-6cc0849233bd response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
24102024-09-09T17:55:53.564ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63510
24112024-09-09T17:55:53.564ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63510 req_id = 7b411243-03e3-4967-b78a-d3f6aa115c65 response_code = 200 uri = /vpc-routes
24122024-09-09T17:55:53.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 193 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63510 req_id = 848de329-1b83-4358-a6f9-a739cd52f591 response_code = 204 uri = /vpc-routes
24132024-09-09T17:55:53.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63839
24142024-09-09T17:55:53.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63839 req_id = c86ca09d-0669-40c8-9017-bdd4228a8d04 response_code = 200 uri = /vpc-routes
24152024-09-09T17:55:53.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39487
24162024-09-09T17:55:53.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 972 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39487 req_id = d0c76419-8d2b-4723-958b-f9a58bfe915f response_code = 200 uri = /v2p
24172024-09-09T17:55:53.734ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63839 req_id = 2b01e116-5126-4766-a9ff-7f5fb59fc990 response_code = 204 uri = /vpc-routes
24182024-09-09T17:55:53.820ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35096
24192024-09-09T17:55:53.820ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24202024-09-09T17:55:53.866ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24212024-09-09T17:55:53.929ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109266 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35096 req_id = d2b23b3e-86cd-4e06-bc3b-3388e7c5bbef response_code = 204 uri = /switch-ports
24222024-09-09T17:55:53.929ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62994
24232024-09-09T17:55:53.929ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52243
24242024-09-09T17:55:53.929ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24252024-09-09T17:55:53.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 293 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35096 req_id = 7b40c8ce-a67a-4ddf-97b3-d57bfbfa9329 response_code = 200 uri = /network-bootstore-config
24262024-09-09T17:55:53.934ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4831 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52243 req_id = e01b0e95-5f1e-4f86-9337-2dd76a33b707 response_code = 200 uri = /v2p
24272024-09-09T17:55:53.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65065
24282024-09-09T17:55:53.960ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65065 req_id = 639f4766-3efb-4955-88e6-fbdc81adad80 response_code = 200 uri = /vpc-routes
24292024-09-09T17:55:53.968ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24302024-09-09T17:55:53.974ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65065 req_id = 06a88910-0ad6-485e-9db2-b081d93fcd40 response_code = 204 uri = /vpc-routes
24312024-09-09T17:55:54.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105648 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62994 req_id = eeafeb69-6786-4371-8eec-2d3e91e83eed response_code = 204 uri = /switch-ports
24322024-09-09T17:55:54.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62994 req_id = b98f6190-8b8a-428e-979a-c0e2a18769b7 response_code = 200 uri = /network-bootstore-config
24332024-09-09T17:55:54.053ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47979
24342024-09-09T17:55:54.054ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47979 req_id = 1f10af57-3912-4d85-ad07-9012c60a0fc7 response_code = 200 uri = /v2p
24352024-09-09T17:55:54.065ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54976
24362024-09-09T17:55:54.065ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24372024-09-09T17:55:54.102ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24382024-09-09T17:55:54.159ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94114 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54976 req_id = 8d449b2f-ceeb-43df-9ab4-a8d3784eec17 response_code = 204 uri = /switch-ports
24392024-09-09T17:55:54.160ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 703 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54976 req_id = c1e681f5-0522-4140-9691-f5099747bb57 response_code = 200 uri = /network-bootstore-config
24402024-09-09T17:55:58.005ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34073
24412024-09-09T17:55:58.005ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34073 req_id = 0e63e5f6-eec4-457a-a979-124c8c874cf4 response_code = 200 uri = /baseboard
24422024-09-09T17:56:01.887ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8034 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 1b90511b-699a-47a4-91dd-9b91d10d0759 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
24432024-09-09T17:56:23.794ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49850
24442024-09-09T17:56:23.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49850 req_id = 830bcf8e-fa49-477f-acfb-b3c7d199bc39 response_code = 200 uri = /vpc-routes
24452024-09-09T17:56:23.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49850 req_id = 91342dd6-7f80-4ddf-a1b6-7bb37cf7d6bc response_code = 204 uri = /vpc-routes
24462024-09-09T17:56:23.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33108
24472024-09-09T17:56:23.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33108 req_id = a82e2691-dc2f-4c1e-82f9-2039da552c7e response_code = 200 uri = /v2p
24482024-09-09T17:56:23.925ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57149
24492024-09-09T17:56:23.925ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57149 req_id = 250c7ba8-444d-43b1-b3f2-e8892d637180 response_code = 200 uri = /v2p
24502024-09-09T17:56:23.938ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55310
24512024-09-09T17:56:23.938ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55310 req_id = 299dc242-133f-4552-b598-ff62da39202b response_code = 200 uri = /vpc-routes
24522024-09-09T17:56:23.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55310 req_id = c4c5a47a-ea5c-4de8-9af5-31576ce2a959 response_code = 204 uri = /vpc-routes
24532024-09-09T17:56:24.016ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57207
24542024-09-09T17:56:24.016ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24552024-09-09T17:56:24.062ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24562024-09-09T17:56:24.129ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112683 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57207 req_id = 9d9fe229-c53c-4ca4-a4b3-f04be4a6d9c4 response_code = 204 uri = /switch-ports
24572024-09-09T17:56:24.129ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48878
24582024-09-09T17:56:24.129ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37115
24592024-09-09T17:56:24.129ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37115 req_id = 63d6990c-8e69-40a0-8041-9735a41ab910 response_code = 200 uri = /vpc-routes
24602024-09-09T17:56:24.130ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24612024-09-09T17:56:24.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1311 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57207 req_id = 203c8122-dde9-496f-ab8b-1e019f73e8b4 response_code = 200 uri = /network-bootstore-config
24622024-09-09T17:56:24.138ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 191 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37115 req_id = 2ed24d8d-378a-44b8-8fcd-912e97573331 response_code = 204 uri = /vpc-routes
24632024-09-09T17:56:24.169ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24642024-09-09T17:56:24.226ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96849 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48878 req_id = f722d344-574a-4546-a71d-708cfd6026b0 response_code = 204 uri = /switch-ports
24652024-09-09T17:56:24.226ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48878 req_id = f2104cc3-dfbd-4a23-b07c-0d37e7832eff response_code = 200 uri = /network-bootstore-config
24662024-09-09T17:56:24.295ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33654
24672024-09-09T17:56:24.295ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33654 req_id = 70564603-9ced-4028-b2c5-689332604ade response_code = 200 uri = /v2p
24682024-09-09T17:56:24.390ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54863
24692024-09-09T17:56:24.390ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24702024-09-09T17:56:24.429ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24712024-09-09T17:56:24.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96908 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54863 req_id = 08a9cc1a-379b-4648-b1a4-821b3184c4a3 response_code = 204 uri = /switch-ports
24722024-09-09T17:56:24.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54863 req_id = fb82710e-136b-401b-95f8-881299a52b7f response_code = 200 uri = /network-bootstore-config
24732024-09-09T17:56:28.038ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41340
24742024-09-09T17:56:28.038ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41340 req_id = af58dea2-2f8c-4163-92fd-f1cd02fb3444 response_code = 200 uri = /baseboard
24752024-09-09T17:56:31.886ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6656 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = b41f2922-66a4-45b6-aa51-34f7b120ec87 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
24762024-09-09T17:56:53.629ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33825
24772024-09-09T17:56:53.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33825 req_id = e1c051da-0f08-417e-9539-ac5121b8f526 response_code = 200 uri = /vpc-routes
24782024-09-09T17:56:53.638ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33825 req_id = 3bba7e48-5fa5-4b7c-bdc6-52300ea13a5b response_code = 204 uri = /vpc-routes
24792024-09-09T17:56:53.719ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64159
24802024-09-09T17:56:53.719ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64159 req_id = ba07ec1f-76e5-40bf-ab06-b9c483c06706 response_code = 200 uri = /vpc-routes
24812024-09-09T17:56:53.729ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64159 req_id = 5dc0b579-9a81-4b4f-b726-8722e172236a response_code = 204 uri = /vpc-routes
24822024-09-09T17:56:53.759ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39564
24832024-09-09T17:56:53.759ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24842024-09-09T17:56:53.807ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24852024-09-09T17:56:53.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 122891 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39564 req_id = 17701f54-29b4-44ec-99e2-c9d01be59873 response_code = 204 uri = /switch-ports
24862024-09-09T17:56:53.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37197
24872024-09-09T17:56:53.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43294
24882024-09-09T17:56:53.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63138
24892024-09-09T17:56:53.883ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24902024-09-09T17:56:53.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4024 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63138 req_id = 6f091ab5-9b6e-42fa-ba57-68e13cdb0562 response_code = 200 uri = /v2p
24912024-09-09T17:56:53.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4062 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43294 req_id = c815dfb9-82a1-4ded-8012-34265b53ba48 response_code = 200 uri = /v2p
24922024-09-09T17:56:53.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4411 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39564 req_id = ac6e563b-5470-45d0-b890-6a8ea75dcc3a response_code = 200 uri = /network-bootstore-config
24932024-09-09T17:56:53.920ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24942024-09-09T17:56:53.977ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94617 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37197 req_id = fd0d216f-fdb6-47b4-800c-ef7c7e5620f4 response_code = 204 uri = /switch-ports
24952024-09-09T17:56:53.977ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37197 req_id = ba50b199-f457-4252-89a4-eba43b8e2563 response_code = 200 uri = /network-bootstore-config
24962024-09-09T17:56:53.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59128
24972024-09-09T17:56:53.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 721 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59128 req_id = db371ecc-a438-459e-98fd-5dc7ab398764 response_code = 200 uri = /vpc-routes
24982024-09-09T17:56:54.022ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63127
24992024-09-09T17:56:54.022ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63127 req_id = 335256e7-60cb-4171-906e-8dadc7268e4c response_code = 200 uri = /v2p
25002024-09-09T17:56:54.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59128 req_id = ffb58abd-6aeb-4da7-bf0e-4c104ce6d3aa response_code = 204 uri = /vpc-routes
25012024-09-09T17:56:54.160ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39384
25022024-09-09T17:56:54.160ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25032024-09-09T17:56:54.199ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25042024-09-09T17:56:54.256ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95759 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39384 req_id = 7ad7b99f-784c-494d-86e6-f086a02cd05a response_code = 204 uri = /switch-ports
25052024-09-09T17:56:54.256ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39384 req_id = 271c9ba3-206b-4dad-94fd-2997be66c77b response_code = 200 uri = /network-bootstore-config
25062024-09-09T17:56:58.072ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52764
25072024-09-09T17:56:58.072ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52764 req_id = 55ca90e8-925c-4284-ae13-17a46b8d47aa response_code = 200 uri = /baseboard
25082024-09-09T17:57:01.887ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7121 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 368eac54-32d4-4bc4-8d43-6f8793fcaced response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
25092024-09-09T17:57:23.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43452
25102024-09-09T17:57:23.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43452 req_id = 458b9b8d-145b-4955-b2b2-7b906f2c2962 response_code = 200 uri = /vpc-routes
25112024-09-09T17:57:23.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43452 req_id = ff36bc25-b014-4351-b4b4-e3a672f48c7f response_code = 204 uri = /vpc-routes
25122024-09-09T17:57:23.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65169
25132024-09-09T17:57:23.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65169 req_id = 64e48ebd-2db9-47eb-87e6-4fcf89c56be4 response_code = 200 uri = /vpc-routes
25142024-09-09T17:57:23.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1180 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:65169 req_id = 9eef1d7c-4164-4c96-a192-747a09f60728 response_code = 204 uri = /vpc-routes
25152024-09-09T17:57:23.898ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57973
25162024-09-09T17:57:23.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57973 req_id = 11a64ab2-20a9-445a-963e-c8310aaa8d7c response_code = 200 uri = /v2p
25172024-09-09T17:57:23.961ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63025
25182024-09-09T17:57:23.961ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25192024-09-09T17:57:24.003ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25202024-09-09T17:57:24.064ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103079 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63025 req_id = 4d4f1951-186c-43da-b7b3-2265c89a9708 response_code = 204 uri = /switch-ports
25212024-09-09T17:57:24.064ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64534
25222024-09-09T17:57:24.064ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64661
25232024-09-09T17:57:24.064ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51878
25242024-09-09T17:57:24.064ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64534 req_id = 86e25953-de8d-4373-a404-9c20de6fa445 response_code = 200 uri = /vpc-routes
25252024-09-09T17:57:24.064ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 368 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63025 req_id = aefd5927-bb71-48e7-a1b8-2be7d38a6ad1 response_code = 200 uri = /network-bootstore-config
25262024-09-09T17:57:24.065ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 323 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51878 req_id = 921b6064-a515-4183-96c4-c32a65799fbc response_code = 200 uri = /v2p
25272024-09-09T17:57:24.065ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25282024-09-09T17:57:24.065ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
25292024-09-09T17:57:24.065ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25302024-09-09T17:57:24.065ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25312024-09-09T17:57:24.065ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25322024-09-09T17:57:24.065ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25332024-09-09T17:57:24.065ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25342024-09-09T17:57:24.065ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 552 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64661 req_id = 62e27a2b-416b-4207-a1b5-186b67aac206 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25352024-09-09T17:57:24.081ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43446
25362024-09-09T17:57:24.082ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25372024-09-09T17:57:24.123ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25382024-09-09T17:57:24.184ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102222 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43446 req_id = 94cee6fb-438b-43b6-9eb9-a1ba91eb00d7 response_code = 204 uri = /switch-ports
25392024-09-09T17:57:24.184ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 293 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43446 req_id = 19c496dd-a30d-4cb6-81ad-3d6b62add3a0 response_code = 200 uri = /network-bootstore-config
25402024-09-09T17:57:24.195ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35725
25412024-09-09T17:57:24.196ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25422024-09-09T17:57:24.196ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
25432024-09-09T17:57:24.196ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25442024-09-09T17:57:24.197ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25452024-09-09T17:57:24.197ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25462024-09-09T17:57:24.197ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25472024-09-09T17:57:24.197ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25482024-09-09T17:57:24.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1064 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35725 req_id = 33978667-75f1-419f-92d4-6a13e7cd32a7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25492024-09-09T17:57:24.220ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55683
25502024-09-09T17:57:24.220ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55683 req_id = 501ae593-002f-4bbb-9c36-c8d1d9b0495b response_code = 200 uri = /v2p
25512024-09-09T17:57:24.226ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64534 req_id = 8ae8f7b2-9661-4157-a072-082de0bde9d8 response_code = 204 uri = /vpc-routes
25522024-09-09T17:57:24.305ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65246
25532024-09-09T17:57:24.305ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25542024-09-09T17:57:24.347ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25552024-09-09T17:57:24.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102051 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65246 req_id = efe4affc-2c2e-47a3-bb22-cb5044551a5c response_code = 204 uri = /switch-ports
25562024-09-09T17:57:24.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65246 req_id = eb46dff0-3a59-423a-8f54-a2c4d685a4c2 response_code = 200 uri = /network-bootstore-config
25572024-09-09T17:57:24.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38177
25582024-09-09T17:57:24.441ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5db99faa-847d-4c67-baad-c7351bdbf8f8, kind: Service { id: 07c1bbdd-f080-4837-9528-41234be47bda }, name: Name("external-dns-07c1bbdd-f080-4837-9528-41234be47bda"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 94])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffc9837e-602d-49be-aea3-1a5919d80850, kind: Service { id: 9711344c-11c4-490b-9560-e9c007eb0cf3 }, name: Name("external-dns-9711344c-11c4-490b-9560-e9c007eb0cf3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 201, 216])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2b31f448-70f4-4ab8-9cd0-36ff4c28255b, kind: Service { id: ca022d43-3da4-4507-941c-6fd9da5b668d }, name: Name("nexus-ca022d43-3da4-4507-941c-6fd9da5b668d"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 185, 185])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 49852929-112c-4ccc-bc3e-c924ce561472, kind: Service { id: fd75f244-5e91-4f63-b3b2-ae4139e91823 }, name: Name("nexus-fd75f244-5e91-4f63-b3b2-ae4139e91823"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 139, 64])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f29fffc4-b4e4-4069-af7f-4a0100b36168, kind: Service { id: 4c6ef9f1-4001-4c65-ae99-bafe915b66f8 }, name: Name("nexus-4c6ef9f1-4001-4c65-ae99-bafe915b66f8"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 44])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25592024-09-09T17:57:24.442ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
25602024-09-09T17:57:24.442ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25612024-09-09T17:57:24.442ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25622024-09-09T17:57:24.442ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25632024-09-09T17:57:24.442ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25642024-09-09T17:57:24.442ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25652024-09-09T17:57:24.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 446 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38177 req_id = e4391d5c-8309-43f2-98b4-8e43335afc2e response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25662024-09-09T17:57:28.108ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55886
25672024-09-09T17:57:28.108ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55886 req_id = fa07c563-ed7a-40f1-8e46-85680862cd27 response_code = 200 uri = /baseboard
25682024-09-09T17:57:31.888ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6585 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 4ecc1b06-d0db-4202-9f21-a0ba8bc58a02 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
25692024-09-09T17:57:53.581ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63735
25702024-09-09T17:57:53.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63735 req_id = a5727053-6f0d-401a-997c-b239de844039 response_code = 200 uri = /vpc-routes
25712024-09-09T17:57:53.604ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38281
25722024-09-09T17:57:53.604ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38281 req_id = 75426916-0081-40cb-9ddc-3d58bcf16bc4 response_code = 200 uri = /vpc-routes
25732024-09-09T17:57:53.604ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63735 req_id = d7b648a6-2096-415f-8f35-bc56e1a282c5 response_code = 204 uri = /vpc-routes
25742024-09-09T17:57:53.615ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:38281 req_id = e2dc21dc-19e4-47b2-af99-464dd6cf5912 response_code = 204 uri = /vpc-routes
25752024-09-09T17:57:53.756ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61400
25762024-09-09T17:57:53.756ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61400 req_id = 728d565e-9616-4f17-93b4-272e4a630e2a response_code = 200 uri = /v2p
25772024-09-09T17:57:53.765ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47763
25782024-09-09T17:57:53.766ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 879 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47763 req_id = aa03b52b-b935-4e5b-9c78-bea64a2e92b2 response_code = 200 uri = /v2p
25792024-09-09T17:57:53.844ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51902
25802024-09-09T17:57:53.844ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25812024-09-09T17:57:53.887ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25822024-09-09T17:57:53.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104401 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51902 req_id = 5e842230-03d3-4130-81ed-f6c38647a5ce response_code = 204 uri = /switch-ports
25832024-09-09T17:57:53.949ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62654
25842024-09-09T17:57:53.949ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25852024-09-09T17:57:53.952ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3726 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51902 req_id = b1b11a6e-e06c-4cd8-a389-dee5394318b7 response_code = 200 uri = /network-bootstore-config
25862024-09-09T17:57:53.984ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25872024-09-09T17:57:53.994ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47865
25882024-09-09T17:57:53.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47865 req_id = 42bcac0e-db8a-4974-8750-044af817a5ff response_code = 200 uri = /vpc-routes
25892024-09-09T17:57:54.003ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47865 req_id = 264345cc-9779-4d31-85af-101339951cb8 response_code = 204 uri = /vpc-routes
25902024-09-09T17:57:54.024ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58827
25912024-09-09T17:57:54.024ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 330 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58827 req_id = 52f2bb9a-e2cb-402c-b60f-e20bc013cc53 response_code = 200 uri = /v2p
25922024-09-09T17:57:54.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95308 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62654 req_id = 4f873aaf-c6fc-40d4-a65e-2c3b490213c5 response_code = 204 uri = /switch-ports
25932024-09-09T17:57:54.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 166 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62654 req_id = af39b25d-59df-418a-adc6-03215f0cb5c2 response_code = 200 uri = /network-bootstore-config
25942024-09-09T17:57:54.141ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64272
25952024-09-09T17:57:54.142ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25962024-09-09T17:57:54.183ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25972024-09-09T17:57:54.240ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99238 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:64272 req_id = 46e100a6-a573-497d-a10d-91d2114d4f32 response_code = 204 uri = /switch-ports
25982024-09-09T17:57:54.241ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64272 req_id = 874a5ab1-6913-4b87-9396-7ca4431c0464 response_code = 200 uri = /network-bootstore-config
25992024-09-09T17:57:58.144ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55729
26002024-09-09T17:57:58.144ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55729 req_id = 488f02ec-f48f-428e-a45c-153e4e6c1dc3 response_code = 200 uri = /baseboard
26012024-09-09T17:58:01.997ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7077 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 7beecdcb-991c-4045-bdff-876dba5edcc1 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
26022024-09-09T17:58:23.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32785
26032024-09-09T17:58:23.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32785 req_id = daa10b92-eba0-4d40-b308-74702638d305 response_code = 200 uri = /vpc-routes
26042024-09-09T17:58:23.809ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:32785 req_id = a933f81f-afbf-461a-8866-f9d3fa859594 response_code = 204 uri = /vpc-routes
26052024-09-09T17:58:23.935ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50810
26062024-09-09T17:58:23.936ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50810 req_id = 5a504dfe-9fb1-4d66-91e2-45a17812136a response_code = 200 uri = /vpc-routes
26072024-09-09T17:58:23.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50810 req_id = a57e55b9-7aea-4a41-a65b-d36f7616f349 response_code = 204 uri = /vpc-routes
26082024-09-09T17:58:23.970ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46051
26092024-09-09T17:58:23.971ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 821 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46051 req_id = abb387ba-3ac3-4e92-80fb-ae539822b953 response_code = 200 uri = /v2p
26102024-09-09T17:58:23.981ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54585
26112024-09-09T17:58:23.981ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54585 req_id = ed5cb17b-9f37-46d4-ae2e-140e390fd378 response_code = 200 uri = /v2p
26122024-09-09T17:58:23.999ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62420
26132024-09-09T17:58:24.000ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26142024-09-09T17:58:24.043ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26152024-09-09T17:58:24.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103022 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 2589b45f-7bee-410e-a4bc-0c7803039ec6 response_code = 204 uri = /switch-ports
26162024-09-09T17:58:24.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47318
26172024-09-09T17:58:24.103ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26182024-09-09T17:58:24.107ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3734 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 3b0452e1-10da-46f0-99b8-a9599035bb05 response_code = 200 uri = /network-bootstore-config
26192024-09-09T17:58:24.138ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26202024-09-09T17:58:24.191ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61609
26212024-09-09T17:58:24.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88864 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47318 req_id = 4ff3f272-4ea4-449c-a3c6-f16f7400bc3b response_code = 204 uri = /switch-ports
26222024-09-09T17:58:24.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61609 req_id = 0f32585d-897d-4524-9e5c-868739938160 response_code = 200 uri = /vpc-routes
26232024-09-09T17:58:24.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47318 req_id = bd22f502-0f70-4af3-aaf9-a77deb637755 response_code = 200 uri = /network-bootstore-config
26242024-09-09T17:58:24.199ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55800
26252024-09-09T17:58:24.199ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55800 req_id = 8f397d47-c2de-4aed-bef0-e55ff310a439 response_code = 200 uri = /v2p
26262024-09-09T17:58:24.201ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 763 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61609 req_id = 609c117c-d280-4c47-b88c-31aa2626f315 response_code = 204 uri = /vpc-routes
26272024-09-09T17:58:24.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40757
26282024-09-09T17:58:24.322ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26292024-09-09T17:58:24.360ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26302024-09-09T17:58:24.413ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91678 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40757 req_id = a3180c4f-c222-4c86-8bc7-115f19982bf4 response_code = 204 uri = /switch-ports
26312024-09-09T17:58:24.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40757 req_id = 43ea2b0f-bf9f-46e9-992f-2a42edb0f000 response_code = 200 uri = /network-bootstore-config
26322024-09-09T17:58:28.179ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36279
26332024-09-09T17:58:28.179ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36279 req_id = ac862bf0-c0de-43bd-9095-11c4e5ecb7d9 response_code = 200 uri = /baseboard
26342024-09-09T17:58:32.001ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7237 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = a6f94761-17cb-4924-9c16-45a05262be61 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
26352024-09-09T17:58:53.668ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62645
26362024-09-09T17:58:53.668ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 887 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62645 req_id = f04590a6-b7a4-4e95-a7ac-1fdc42f2d61a response_code = 200 uri = /vpc-routes
26372024-09-09T17:58:53.678ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62645 req_id = 7370d927-1cb2-4d40-9d49-b442a8064a6b response_code = 204 uri = /vpc-routes
26382024-09-09T17:58:53.685ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42924
26392024-09-09T17:58:53.685ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42924 req_id = e20b01bb-8e7a-4455-8dcd-b59dfc4e9ebf response_code = 200 uri = /vpc-routes
26402024-09-09T17:58:53.694ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:42924 req_id = 6563b99c-2955-457f-913b-4eaf1a4404ee response_code = 204 uri = /vpc-routes
26412024-09-09T17:58:53.741ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42887
26422024-09-09T17:58:53.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42887 req_id = 18d566e3-9d28-4356-8f2b-3479b6dd37e9 response_code = 200 uri = /v2p
26432024-09-09T17:58:53.775ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39038
26442024-09-09T17:58:53.775ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26452024-09-09T17:58:53.818ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26462024-09-09T17:58:53.878ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102780 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39038 req_id = 5d5a2c7a-6013-4a05-aaa8-44bde781639b response_code = 204 uri = /switch-ports
26472024-09-09T17:58:53.878ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53577
26482024-09-09T17:58:53.878ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39723
26492024-09-09T17:58:53.878ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26502024-09-09T17:58:53.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4414 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39038 req_id = bc5b8ab3-6d3e-4146-be03-c337769a81c3 response_code = 200 uri = /network-bootstore-config
26512024-09-09T17:58:53.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4681 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39723 req_id = 5f7aade3-e984-41d1-be70-0b470400e007 response_code = 200 uri = /v2p
26522024-09-09T17:58:53.913ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26532024-09-09T17:58:53.916ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42988
26542024-09-09T17:58:53.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 204 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42988 req_id = 245a8807-1d66-4d8a-943c-fc0ad4bbe976 response_code = 200 uri = /vpc-routes
26552024-09-09T17:58:53.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1084 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42988 req_id = 29e0d2d1-dea0-4629-9983-eed099aa5adf response_code = 204 uri = /vpc-routes
26562024-09-09T17:58:53.973ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94798 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:53577 req_id = 5bdbb042-4a1f-4ffe-8df4-79af73ad49df response_code = 204 uri = /switch-ports
26572024-09-09T17:58:53.973ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53577 req_id = 2c5dc200-7346-476d-b51e-1f362a8726db response_code = 200 uri = /network-bootstore-config
26582024-09-09T17:58:54.081ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58302
26592024-09-09T17:58:54.081ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58302 req_id = 8003b7cb-f8f6-4cbf-94a9-03faaaab2f9b response_code = 200 uri = /v2p
26602024-09-09T17:58:54.146ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59855
26612024-09-09T17:58:54.146ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26622024-09-09T17:58:54.181ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26632024-09-09T17:58:54.238ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91737 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59855 req_id = febd7cb0-86d8-42cb-9a94-5980bdc14791 response_code = 204 uri = /switch-ports
26642024-09-09T17:58:54.238ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59855 req_id = 06324f1d-e1a6-4453-8320-f0731e8eb3b7 response_code = 200 uri = /network-bootstore-config
26652024-09-09T17:58:58.217ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53205
26662024-09-09T17:58:58.218ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 801 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53205 req_id = cb40750a-ccda-4ead-9bea-c75dd5b968a0 response_code = 200 uri = /baseboard
26672024-09-09T17:58:58.798ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34610
26682024-09-09T17:58:58.798ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34610 req_id = f2f9cae3-0196-479e-9165-e59bd32e17d8 response_code = 200 uri = /v2p
26692024-09-09T17:58:58.802ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(6460566) }
26702024-09-09T17:58:58.804ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5696 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34610 req_id = ef3bc583-6862-4edb-bc9e-f438cbb945c6 response_code = 204 uri = /v2p
26712024-09-09T17:58:58.968ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61451
26722024-09-09T17:58:58.994ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:570 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(6460566))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-eaf7a0b5a9d", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 3d51e186-2159-48c3-bf34-c47f7cf6e34c, read_only_parent: Some(Volume { block_size: 512, id: 4ef107b1-d950-4fe9-9175-980b859ce150, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: d09c4ac9-5522-462d-a22f-856a2751a875, key: Some("0AIv4BNGw5iDC/JGKKWaI0ZkTuvWfKNbHYhIEGyFGcQ="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3d51e186-2159-48c3-bf34-c47f7cf6e34c, key: Some("s9ODaR7tDM7jnWQ8kxA+DktuC+UtBusc5QVRpfmhDeE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some(..) } instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 instance_runtime = InstanceRuntimeState { propolis_id: Some(760054d9-c981-48b3-88ba-f2e26d33bb6c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-09-09T17:58:58.539917206Z } metadata = InstanceMetadata { silo_id: 8dd86d6f-ad2d-45da-852c-8351c18c2cc5, project_id: 7f64605e-b035-408a-9a6a-d8f7d5f3252b } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-09-09T17:58:58.496726Z }
26732024-09-09T17:58:58.994ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:603 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26742024-09-09T17:58:58.996ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:991 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 migration_id = None propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(6460566))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21680f30-0060-42cd-8e6b-66d832fccd47, kind: Instance { id: 9a69ee61-cd07-4e16-8bce-71a799ee7a11 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 208, 226])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(6460566), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-eaf7a0b5a9d", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 3d51e186-2159-48c3-bf34-c47f7cf6e34c, read_only_parent: Some(Volume { block_size: 512, id: 4ef107b1-d950-4fe9-9175-980b859ce150, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: d09c4ac9-5522-462d-a22f-856a2751a875, key: Some("0AIv4BNGw5iDC/JGKKWaI0ZkTuvWfKNbHYhIEGyFGcQ="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3d51e186-2159-48c3-bf34-c47f7cf6e34c, key: Some("s9ODaR7tDM7jnWQ8kxA+DktuC+UtBusc5QVRpfmhDeE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-09-09T17:58:58.496726Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
26752024-09-09T17:58:58.998ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 30334 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61451 req_id = cb5ea5d2-751b-49c9-8a9a-a8860c7d1969 response_code = 200 uri = /vmms/760054d9-c981-48b3-88ba-f2e26d33bb6c
26762024-09-09T17:58:59.111ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59010
26772024-09-09T17:58:59.114ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2533 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59010 req_id = f8cf17e7-475e-41ef-8f95-e1a1c0d13a65 response_code = 200 uri = /vmms/760054d9-c981-48b3-88ba-f2e26d33bb6c/state
26782024-09-09T17:58:59.120ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 254, 208, 226]), slot: 0, vni: Vni { inner: 6460566 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
26792024-09-09T17:58:59.121ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
26802024-09-09T17:58:59.162ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c file = illumos-utils/src/zone.rs:316 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26812024-09-09T17:58:59.186ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c file = illumos-utils/src/zone.rs:354 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26822024-09-09T17:59:00.356ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26832024-09-09T17:59:00.357ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c zone file = illumos-utils/src/running_zone.rs:488 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26842024-09-09T17:59:00.552ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 33.149749ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26852024-09-09T17:59:00.594ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 130.874487ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26862024-09-09T17:59:00.743ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 210.521036ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26872024-09-09T17:59:00.969ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 320.24397ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26882024-09-09T17:59:01.307ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 413.756949ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26892024-09-09T17:59:01.732ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 1.37847244s file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26902024-09-09T17:59:02.001ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6778 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = 8d6f9da7-395d-451e-96bc-499b7b83ac78 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
26912024-09-09T17:59:03.123ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 705.026674ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26922024-09-09T17:59:03.840ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 1.203464365s file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26932024-09-09T17:59:05.055ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 599.668673ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26942024-09-09T17:59:05.665ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 634.407902ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c zone = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
26952024-09-09T17:59:06.328ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c file = sled-agent/src/instance.rs:1449 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26962024-09-09T17:59:06.337ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 64.372682ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26972024-09-09T17:59:06.412ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 136.621199ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26982024-09-09T17:59:06.558ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 173.500049ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
26992024-09-09T17:59:06.741ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c") failed: Property not found. retry in 405.660709ms file = illumos-utils/src/svc.rs:36 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27002024-09-09T17:59:07.158ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1458 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27012024-09-09T17:59:07.196ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1482 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27022024-09-09T17:59:07.339ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:727 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27032024-09-09T17:59:07.343ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-09-09T17:59:07.343884227Z } file = sled-agent/src/instance.rs:594 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27042024-09-09T17:59:07.344ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-09T17:59:07.343884227Z } propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27052024-09-09T17:59:07.345ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-09T17:59:07.343884227Z }, migration_in: None, migration_out: None }
27062024-09-09T17:59:07.506ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:197
27072024-09-09T17:59:07.506ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
27082024-09-09T17:59:07.562ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-09-09T17:59:07.562006285Z } file = sled-agent/src/instance.rs:594 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27092024-09-09T17:59:07.562ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-09-09T17:59:07.562006285Z } propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27102024-09-09T17:59:07.562ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-09-09T17:59:07.562006285Z }, migration_in: None, migration_out: None }
27112024-09-09T17:59:07.603ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 34669, tv_nsec: 200110218 } time_to_next_cleanup = 599.99999576s
27122024-09-09T17:59:22.568ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
27132024-09-09T17:59:23.743ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60852
27142024-09-09T17:59:23.743ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60852 req_id = 5237a9ab-3f48-4b2c-b8d0-a0c3aca22d4b response_code = 200 uri = /vpc-routes
27152024-09-09T17:59:23.797ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33619
27162024-09-09T17:59:23.799ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1796 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33619 req_id = 46f30c23-53db-4f86-a5a2-3696900f2dc4 response_code = 200 uri = /vmms/760054d9-c981-48b3-88ba-f2e26d33bb6c/state
27172024-09-09T17:59:23.806ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50525
27182024-09-09T17:59:23.807ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50525 req_id = 731552fe-0f72-47b0-90e4-da8c5cc9c965 response_code = 200 uri = /vmms/760054d9-c981-48b3-88ba-f2e26d33bb6c/state
27192024-09-09T17:59:23.832ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48191
27202024-09-09T17:59:23.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48191 req_id = 61fad09b-4e5b-413b-8771-6c07d89dde3d response_code = 200 uri = /vpc-routes
27212024-09-09T17:59:23.949ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42867
27222024-09-09T17:59:23.950ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42867 req_id = dcafbee7-7dbf-4385-aed0-00060a18740b response_code = 200 uri = /v2p
27232024-09-09T17:59:23.959ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49150
27242024-09-09T17:59:23.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49150 req_id = 61d3e709-b068-4dcf-aada-f8a71e7792ea response_code = 200 uri = /vmms/760054d9-c981-48b3-88ba-f2e26d33bb6c/state
27252024-09-09T17:59:23.977ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 335 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60852 req_id = 43e07ffd-6e55-45f2-8027-86f1d66f86ea response_code = 204 uri = /vpc-routes
27262024-09-09T17:59:24.043ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49205
27272024-09-09T17:59:24.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49205 req_id = 81ada18c-c657-4cc1-9ac5-4dc5b987faed response_code = 200 uri = /v2p
27282024-09-09T17:59:24.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36951
27292024-09-09T17:59:24.048ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27302024-09-09T17:59:24.090ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27312024-09-09T17:59:24.150ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102790 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36951 req_id = c77d744b-8551-444f-b613-fb73cf183475 response_code = 204 uri = /switch-ports
27322024-09-09T17:59:24.151ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37217
27332024-09-09T17:59:24.151ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 208 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48191 req_id = f9d23242-16d7-42f4-a92e-8acd7481057f response_code = 204 uri = /vpc-routes
27342024-09-09T17:59:24.151ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27352024-09-09T17:59:24.158ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7464 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36951 req_id = a4b7e507-94e5-4fae-b9f2-c5e627ce3c4c response_code = 200 uri = /network-bootstore-config
27362024-09-09T17:59:24.170ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54690
27372024-09-09T17:59:24.171ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 281 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54690 req_id = 467ee50d-b575-475d-90b2-e62e913a1cb2 response_code = 200 uri = /vpc-routes
27382024-09-09T17:59:24.177ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59904
27392024-09-09T17:59:24.177ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59904 req_id = 403b88b6-38c6-4ec7-99ae-78b08fc94910 response_code = 200 uri = /v2p
27402024-09-09T17:59:24.190ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27412024-09-09T17:59:24.195ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 949 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54690 req_id = c377b76d-78fd-438a-9ac0-5a3c001897d0 response_code = 204 uri = /vpc-routes
27422024-09-09T17:59:24.248ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96864 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37217 req_id = 23f8c2d1-2e3b-48e0-a7d1-c704c5ecd53b response_code = 204 uri = /switch-ports
27432024-09-09T17:59:24.248ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37217 req_id = b54662b2-a348-4847-8793-65ec9a62933e response_code = 200 uri = /network-bootstore-config
27442024-09-09T17:59:24.250ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32975
27452024-09-09T17:59:24.251ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27462024-09-09T17:59:24.286ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27472024-09-09T17:59:24.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91455 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:32975 req_id = fe9938f6-0189-4b52-851f-c4fdbafd5a80 response_code = 204 uri = /switch-ports
27482024-09-09T17:59:24.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:32975 req_id = 595d4ac6-37cf-4bf2-8374-8a1dbdf9c74c response_code = 200 uri = /network-bootstore-config
27492024-09-09T17:59:28.253ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51264
27502024-09-09T17:59:28.253ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51264 req_id = bc78e4fd-10bf-4e3b-b612-7231df309ea3 response_code = 200 uri = /baseboard
27512024-09-09T17:59:32.002ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6775 local_addr = [fd00:1122:3344:101::1]:42327 method = GET remote_addr = [fd00:1122:3344:101::d]:47540 req_id = d3ab5b9e-e2f4-4070-a146-e1628287c5a2 response_code = 200 sled_id = 61e07fcc-45b4-4b25-ae0d-e78eee62eb60 uri = /61e07fcc-45b4-4b25-ae0d-e78eee62eb60
27522024-09-09T17:59:32.110ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39409
27532024-09-09T17:59:32.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 715 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39409 req_id = aefea8bd-d4b0-4363-bf27-13daaa75d465 response_code = 200 uri = /vmms/760054d9-c981-48b3-88ba-f2e26d33bb6c/state
27542024-09-09T17:59:32.111ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-09-09T17:59:32.111713153Z } file = sled-agent/src/instance.rs:594 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27552024-09-09T17:59:32.111ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-09-09T17:59:32.111713153Z } propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27562024-09-09T17:59:32.111ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-09-09T17:59:32.111713153Z }, migration_in: None, migration_out: None }
27572024-09-09T17:59:32.158ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-09-09T17:59:32.158078734Z } file = sled-agent/src/instance.rs:594 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27582024-09-09T17:59:32.158ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-09-09T17:59:32.158078734Z } propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27592024-09-09T17:59:32.158ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:630 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27602024-09-09T17:59:32.158ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
27612024-09-09T17:59:32.158ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
27622024-09-09T17:59:32.158ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/6830ec9d-22cb-45c0-ba9d-5a3728bef44f/debug/bundle/zone", "/pool/int/d05e61ff-a727-4851-80ca-eab919525a00/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/e68e093e-0654-4278-a79f-9c234ab4cc35/crypt/debug", "/pool/ext/f5eda459-d1e9-44f5-b731-f6490e3fc719/crypt/debug", "/pool/ext/677df3fa-a1f5-4e63-93ee-592a272d5e3f/crypt/debug", "/pool/ext/89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crypt/debug", "/pool/ext/46b15cce-a870-48ea-aab8-5f90f01b99e4/crypt/debug", "/pool/ext/0720cae8-ac21-4fb2-9b27-8f836e4c5c41/crypt/debug", "/pool/ext/f8e6d791-b5fc-4ef9-b2c8-37d1fd7dd299/crypt/debug", "/pool/ext/2c079939-1543-4f7b-ab38-d421cbf95e07/crypt/debug", "/pool/ext/b6fbc8d5-08ec-4cd5-ae5d-395890772649/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c
27632024-09-09T17:59:32.802ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1137 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c", bundle_id: b4d820d9-b646-47f8-95e3-9e7aa8a2bb6c }, time_created: 2024-09-09T17:59:32.158740556Z, version: 0, cause: TerminatedInstance }
27642024-09-09T17:59:32.802ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_760054d9-c981-48b3-88ba-f2e26d33bb6c file = sled-agent/src/instance.rs:826 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27652024-09-09T17:59:34.553ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27662024-09-09T17:59:34.569ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(6460566), kind: System }
27672024-09-09T17:59:34.569ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(6460566), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
27682024-09-09T17:59:34.570ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-09-09T17:59:34.570863819Z }, migration_in: None, migration_out: None }
27692024-09-09T17:59:34.600ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-09-09T17:59:34.570863819Z }, migration_in: None, migration_out: None }
27702024-09-09T17:59:34.605ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:763 instance_id = 9a69ee61-cd07-4e16-8bce-71a799ee7a11 propolis_id = 760054d9-c981-48b3-88ba-f2e26d33bb6c
27712024-09-09T17:59:35.121ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54018
27722024-09-09T17:59:35.122ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54018 req_id = 9524703b-75c3-421f-9a7d-55816ffd5f49 response_code = 200 uri = /vpc-routes
27732024-09-09T17:59:35.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54018 req_id = 40c3d96d-a1a7-42f5-877a-9bad820f7720 response_code = 204 uri = /vpc-routes
27742024-09-09T17:59:35.250ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41452
27752024-09-09T17:59:35.250ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41452 req_id = 734f852e-67fc-4283-bd9a-87b617107673 response_code = 200 uri = /v2p
27762024-09-09T17:59:35.619ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50099
27772024-09-09T17:59:35.619ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50099 req_id = c922ee82-4ff7-48a7-9c34-2e4b2d2ceb98 response_code = 200 uri = /v2p
27782024-09-09T17:59:36.651ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61443
27792024-09-09T17:59:36.652ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 766 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61443 req_id = 11e086ce-5052-45b8-9327-60d6b6fcb8e3 response_code = 200 uri = /vpc-routes
27802024-09-09T17:59:36.660ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61443 req_id = 742872e1-9d2e-4af1-9757-71b3d6367817 response_code = 204 uri = /vpc-routes