Max level shown:
1 [ Dec 6 05:03:52 Enabled. ]
2 [ Dec 6 05:03:52 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Dec 6 05:03:52 Method "start" exited with status 0. ]
4 [ Dec 6 05:03:52 Rereading configuration. ]
5 [ Dec 6 05:03:52 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72023-12-06T05:03:52.769ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:240
82023-12-06T05:03:52.824ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:247
92023-12-06T05:03:52.824ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
102023-12-06T05:03:52.937ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
112023-12-06T05:03:52.946ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
122023-12-06T05:03:52.954ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:257 path = /var/run/oxide/
132023-12-06T05:03:53.562ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:a02a::, len: 64 }
142023-12-06T05:03:53.703ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:117
152023-12-06T05:03:53.703ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:129
162023-12-06T05:03:53.703ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
172023-12-06T05:03:53.703ZINFOSledAgent (DumpSetup-archival): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:171
182023-12-06T05:03:53.703ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 sled_mode = Scrimlet { asic: SoftNpuZone }
192023-12-06T05:03:53.703ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
202023-12-06T05:03:53.703ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:606
212023-12-06T05:03:53.706ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "000c3dd0fcba4d4493e687d9695a9876", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:532
222023-12-06T05:03:53.706ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:655
232023-12-06T05:03:53.706ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
242023-12-06T05:03:53.706ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:713
252023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b file = sled-agent/src/long_running_tasks.rs:232
262023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b file = sled-agent/src/long_running_tasks.rs:232
272023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b file = sled-agent/src/long_running_tasks.rs:232
282023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
292023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
302023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
312023-12-06T05:03:53.706ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:241 underlay_network_info = None
322023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
332023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 file = sled-agent/src/long_running_tasks.rs:232
342023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 file = sled-agent/src/long_running_tasks.rs:232
352023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_31bd71cd-4736-4a12-a387-9b74b050396f file = sled-agent/src/long_running_tasks.rs:232
362023-12-06T05:03:53.706ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 file = sled-agent/src/long_running_tasks.rs:232
372023-12-06T05:03:53.706ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:88
382023-12-06T05:03:53.706ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } })) file = sled-storage/src/manager.rs:346
392023-12-06T05:03:53.897ZINFOSledAgent (StorageManager): Channel capacity decreased current = 245 file = sled-storage/src/manager.rs:326 previous = 256
402023-12-06T05:03:53.899ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } file = sled-agent/src/long_running_tasks.rs:90
412023-12-06T05:03:53.899ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } })) file = sled-storage/src/manager.rs:346
422023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External } })) file = sled-storage/src/manager.rs:346
432023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
442023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
452023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
462023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
472023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" }, zpool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } })) file = sled-storage/src/manager.rs:346
482023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" }, zpool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External } })) file = sled-storage/src/manager.rs:346
492023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" }, zpool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } })) file = sled-storage/src/manager.rs:346
502023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" }, zpool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } })) file = sled-storage/src/manager.rs:346
512023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received DisksChanged({}) file = sled-storage/src/manager.rs:346
522023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Channel capacity increased current = 255 file = sled-storage/src/manager.rs:336 previous = 245
532023-12-06T05:03:54.074ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
542023-12-06T05:03:54.074ZINFOSledAgent (StorageMonitor): Received storage manager update file = sled-agent/src/storage_monitor.rs:111 resources = StorageResources { disks: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 497664, free: 20937467904, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 499200, free: 20937466368, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } })} }
552023-12-06T05:03:54.074ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
562023-12-06T05:03:54.075ZWARNSledAgent (bootstore): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:187 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
572023-12-06T05:03:54.075ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
582023-12-06T05:03:54.075ZWARNSledAgent (bootstore): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:187 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
592023-12-06T05:03:54.075ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
602023-12-06T05:03:54.075ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
612023-12-06T05:03:54.075ZWARNSledAgent (bootstore): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json file = common/src/ledger.rs:187 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
622023-12-06T05:03:54.075ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
632023-12-06T05:03:54.075ZWARNSledAgent (bootstore): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json file = common/src/ledger.rs:187 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
642023-12-06T05:03:54.075ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
652023-12-06T05:03:54.075ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
662023-12-06T05:03:54.075ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
672023-12-06T05:03:54.075ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
682023-12-06T05:03:54.090ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:575
692023-12-06T05:03:54.090ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:288 next_cleanup = Instant { tv_sec: 1671, tv_nsec: 172033659 } time_to_next_cleanup = 599.999950869s
702023-12-06T05:03:54.090ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3042
712023-12-06T05:03:54.090ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
722023-12-06T05:03:54.090ZWARNSledAgent (BootstrapAgentStartup): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json file = common/src/ledger.rs:187
732023-12-06T05:03:54.090ZERROSledAgent (BootstrapAgentStartup): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json
742023-12-06T05:03:54.092ZWARNSledAgent (BootstrapAgentStartup): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json file = common/src/ledger.rs:187
752023-12-06T05:03:54.092ZERROSledAgent (BootstrapAgentStartup): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json
762023-12-06T05:03:54.093ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:195 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
772023-12-06T05:03:54.093ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
782023-12-06T05:03:54.093ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { rack_subnet: fd00:1122:3344:100::, trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: Some( RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:1::, prefix: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.2.100, }, ], addresses: [ V4( Ipv4Network { addr: 10.151.2.174, prefix: 32, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], }, ), } file = sled-agent/src/rack_setup/service.rs:803
792023-12-06T05:03:54.094ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
802023-12-06T05:03:54.094ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
812023-12-06T05:03:54.094ZWARNSledAgent (RSS): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker file = common/src/ledger.rs:187
822023-12-06T05:03:54.094ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker
832023-12-06T05:03:54.094ZWARNSledAgent (RSS): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker file = common/src/ledger.rs:187
842023-12-06T05:03:54.094ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker
852023-12-06T05:03:54.094ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:863
862023-12-06T05:03:54.094ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
872023-12-06T05:03:54.094ZWARNSledAgent (RSS): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json file = common/src/ledger.rs:187
882023-12-06T05:03:54.094ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json
892023-12-06T05:03:54.094ZWARNSledAgent (RSS): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json file = common/src/ledger.rs:187
902023-12-06T05:03:54.094ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json
912023-12-06T05:03:54.094ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:904
922023-12-06T05:03:54.094ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
932023-12-06T05:03:54.094ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
942023-12-06T05:03:54.094ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
952023-12-06T05:03:54.094ZINFOSledAgent (RSS): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:198
962023-12-06T05:03:54.094ZINFOSledAgent (RSS): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:198
972023-12-06T05:03:54.094ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
982023-12-06T05:03:54.094ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:937
992023-12-06T05:03:54.094ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:556 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
1002023-12-06T05:03:54.094ZINFOSledAgent (bootstore): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:198 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
1012023-12-06T05:03:54.095ZINFOSledAgent (bootstore): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:198 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
1022023-12-06T05:03:54.095ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:640 peer_id = pc-000c3dd0fcba4d4493e687d9695a9876-i86pc
1032023-12-06T05:03:54.095ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 81c25e4c-3681-431d-a398-2d4416a62095, rack_id: 9fd18c44-b5ff-4007-bd14-a497b2fbd8ea, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1042023-12-06T05:03:54.095ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:50613
1052023-12-06T05:03:54.095ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 81c25e4c-3681-431d-a398-2d4416a62095, rack_id: 9fd18c44-b5ff-4007-bd14-a497b2fbd8ea, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } file = sled-agent/src/bootstrap/server.rs:364
1062023-12-06T05:03:54.095ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:379
1072023-12-06T05:03:54.095ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:47
1082023-12-06T05:03:54.095ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1092023-12-06T05:03:54.095ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:346
1102023-12-06T05:03:54.095ZINFOSledAgent (StorageManager): Attempting to add queued disks file = sled-storage/src/manager.rs:394 num_disks = 9
1112023-12-06T05:03:54.098ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3270
1122023-12-06T05:03:54.105ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1132023-12-06T05:03:54.105ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1142023-12-06T05:03:54.105ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1152023-12-06T05:03:54.105ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1162023-12-06T05:03:54.105ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1172023-12-06T05:03:54.105ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1182023-12-06T05:03:54.137ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1192023-12-06T05:03:54.141ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
1202023-12-06T05:03:54.146ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1212023-12-06T05:03:54.166ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:305 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
1222023-12-06T05:03:54.166ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1232023-12-06T05:03:54.267ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1242023-12-06T05:03:54.267ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1252023-12-06T05:03:54.267ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1262023-12-06T05:03:54.267ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1272023-12-06T05:03:54.267ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1282023-12-06T05:03:54.267ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1292023-12-06T05:03:54.295ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1302023-12-06T05:03:54.303ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1312023-12-06T05:03:54.426ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1322023-12-06T05:03:54.426ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1332023-12-06T05:03:54.426ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1342023-12-06T05:03:54.426ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1352023-12-06T05:03:54.426ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1362023-12-06T05:03:54.426ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1372023-12-06T05:03:54.454ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1382023-12-06T05:03:54.462ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1392023-12-06T05:03:54.586ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1402023-12-06T05:03:54.586ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1412023-12-06T05:03:54.586ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1422023-12-06T05:03:54.586ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1432023-12-06T05:03:54.586ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1442023-12-06T05:03:54.586ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1452023-12-06T05:03:54.615ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1462023-12-06T05:03:54.623ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone file = sled-storage/src/dataset.rs:323
1472023-12-06T05:03:54.744ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1482023-12-06T05:03:54.744ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1492023-12-06T05:03:54.744ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1502023-12-06T05:03:54.744ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1512023-12-06T05:03:54.744ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1522023-12-06T05:03:54.744ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1532023-12-06T05:03:54.772ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1542023-12-06T05:03:54.781ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone file = sled-storage/src/dataset.rs:323
1552023-12-06T05:03:54.904ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1562023-12-06T05:03:54.904ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1572023-12-06T05:03:54.904ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1582023-12-06T05:03:54.904ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1592023-12-06T05:03:54.904ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1602023-12-06T05:03:54.904ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1612023-12-06T05:03:54.933ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1622023-12-06T05:03:54.941ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone file = sled-storage/src/dataset.rs:323
1632023-12-06T05:03:55.065ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1642023-12-06T05:03:55.065ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1652023-12-06T05:03:55.065ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1662023-12-06T05:03:55.065ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1672023-12-06T05:03:55.065ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1682023-12-06T05:03:55.065ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1692023-12-06T05:03:55.093ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1702023-12-06T05:03:55.101ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone file = sled-storage/src/dataset.rs:323
1712023-12-06T05:03:55.220ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1722023-12-06T05:03:55.220ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1732023-12-06T05:03:55.220ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1742023-12-06T05:03:55.220ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1752023-12-06T05:03:55.220ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1762023-12-06T05:03:55.220ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1772023-12-06T05:03:55.248ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1782023-12-06T05:03:55.255ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone file = sled-storage/src/dataset.rs:323
1792023-12-06T05:03:55.376ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1802023-12-06T05:03:55.376ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1812023-12-06T05:03:55.376ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1822023-12-06T05:03:55.376ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1832023-12-06T05:03:55.376ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1842023-12-06T05:03:55.376ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1852023-12-06T05:03:55.404ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1862023-12-06T05:03:55.411ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1872023-12-06T05:03:55.522ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
1882023-12-06T05:03:55.522ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
1892023-12-06T05:03:55.522ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:317 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
1902023-12-06T05:03:55.527ZINFOSledAgent (StorageMonitor): Received storage manager update file = sled-agent/src/storage_monitor.rs:111 resources = StorageResources { disks: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 775680, free: 20937189888, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 769536, free: 20937196032, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" }, zpool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }), Pool { name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, info: ZpoolInfo { name: "oxp_31bd71cd-4736-4a12-a387-9b74b050396f", size: 20937965568, allocated: 818688, free: 20937146880, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" }, zpool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } }), Pool { name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, info: ZpoolInfo { name: "oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7", size: 20937965568, allocated: 774144, free: 20937191424, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 497664, free: 20937467904, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 499200, free: 20937466368, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" }, zpool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } }), Pool { name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, info: ZpoolInfo { name: "oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930", size: 20937965568, allocated: 774144, free: 20937191424, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" }, zpool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External } }), Pool { name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External }, info: ZpoolInfo { name: "oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60", size: 20937965568, allocated: 772608, free: 20937192960, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External } }), Pool { name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External }, info: ZpoolInfo { name: "oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 777216, free: 20937188352, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 772608, free: 20937192960, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 777216, free: 20937188352, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } })} }
1912023-12-06T05:03:55.545ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1922023-12-06T05:03:55.547ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:314
1932023-12-06T05:03:55.566ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:353
1942023-12-06T05:03:55.568ZINFOSledAgent: successfully created encrypted zvol: name="oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1952023-12-06T05:03:55.568ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/swap" file = sled-agent/src/swap_device.rs:96
1962023-12-06T05:03:55.573ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:332 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
1972023-12-06T05:03:55.573ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1982023-12-06T05:03:55.573ZINFOSledAgent (BackingFs): Ensuring dataset oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/backing/fmd file = sled-agent/src/backing_fs.rs:136
1992023-12-06T05:03:55.639ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
2002023-12-06T05:03:55.857ZINFOSledAgent (BackingFs): Mounting oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
2012023-12-06T05:03:55.868ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
2022023-12-06T05:03:55.869ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
2032023-12-06T05:03:55.869ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
2042023-12-06T05:03:55.869ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
2052023-12-06T05:03:56.039ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:162
2062023-12-06T05:03:56.567ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:80 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
2072023-12-06T05:03:56.568ZINFOSledAgent (StorageMonitor): Underlay Available file = sled-agent/src/storage_monitor.rs:122 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
2082023-12-06T05:03:56.568ZINFOSledAgent (InstanceManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/instance_manager.rs:177
2092023-12-06T05:03:56.568ZINFOSledAgent (InstanceManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/instance_manager.rs:185
2102023-12-06T05:03:56.580ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2112023-12-06T05:03:56.581ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2122023-12-06T05:03:56.581ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/14b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2132023-12-06T05:03:56.581ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/cd70d7f6-2354-4bf2-8012-55bf9eaf7930): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2142023-12-06T05:03:56.581ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/f4b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2152023-12-06T05:03:56.581ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2162023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2172023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2182023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: M2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2192023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: M2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2202023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2212023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2222023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2232023-12-06T05:03:56.582ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2242023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/24b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2252023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_31bd71cd-4736-4a12-a387-9b74b050396f on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/31bd71cd-4736-4a12-a387-9b74b050396f): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2262023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/616b26df-e62a-4c68-b506-f4a923d8aaf7): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2272023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/a462a7f7-b628-40fe-80ff-4e4189e2d62b): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2282023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/b462a7f7-b628-40fe-80ff-4e4189e2d62b): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2292023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/ceb4461c-cf56-4719-ad3c-14430bfdfb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2302023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/d462a7f7-b628-40fe-80ff-4e4189e2d62b): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2312023-12-06T05:03:56.583ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095/zpools/e4b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2322023-12-06T05:03:58.870ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_switch
2332023-12-06T05:03:59.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 70.163226ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2342023-12-06T05:03:59.146ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
2352023-12-06T05:03:59.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 146.989766ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2362023-12-06T05:03:59.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 250.996969ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2372023-12-06T05:03:59.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 499.047213ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2382023-12-06T05:04:00.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 729.493566ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2392023-12-06T05:04:01.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.45511958s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2402023-12-06T05:04:02.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.346198605s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2412023-12-06T05:04:04.147ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
2422023-12-06T05:04:05.270ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:1716
2432023-12-06T05:04:06.683ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:938 underlay_address = fd00:1122:3344:101::1
2442023-12-06T05:04:06.683ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2452023-12-06T05:04:06.744ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::1]:12345
2462023-12-06T05:04:06.744ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
2472023-12-06T05:04:06.744ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:424
2482023-12-06T05:04:06.744ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2492023-12-06T05:04:06.744ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:198
2502023-12-06T05:04:06.744ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 345.195917ms sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
2512023-12-06T05:04:06.744ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:198
2522023-12-06T05:04:06.745ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:137
2532023-12-06T05:04:06.745ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:50613
2542023-12-06T05:04:06.745ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:241 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:1::, prefix: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Network { addr: 0.0.0.0, prefix: 0 }), nexthop: 10.151.2.100 }], addresses: [V4(Ipv4Network { addr: 10.151.2.174, prefix: 32 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [] })))
2552023-12-06T05:04:06.745ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3042
2562023-12-06T05:04:06.745ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2572023-12-06T05:04:06.745ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2582023-12-06T05:04:06.748ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2592023-12-06T05:04:06.748ZWARNSledAgent (RSS): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json file = common/src/ledger.rs:187
2602023-12-06T05:04:06.748ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json
2612023-12-06T05:04:06.748ZWARNSledAgent (RSS): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json file = common/src/ledger.rs:187
2622023-12-06T05:04:06.748ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json
2632023-12-06T05:04:06.748ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2642023-12-06T05:04:06.749ZINFOSledAgent: failed to notify nexus about sled agent error = Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/81c25e4c-3681-431d-a398-2d4416a62095): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/sled_agent.rs:666 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
2652023-12-06T05:04:06.781ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42428
2662023-12-06T05:04:06.781ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2672023-12-06T05:04:06.781ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 192 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42428 req_id = f18130f4-b207-4ba7-84b6-00fcb789bdab response_code = 200 uri = /zpools
2682023-12-06T05:04:06.811ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40727
2692023-12-06T05:04:06.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40727 req_id = 9c203ea1-ffe1-4f69-9072-693934970e3e response_code = 200 uri = /sled-role
2702023-12-06T05:04:06.815ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2712023-12-06T05:04:06.815ZINFOSledAgent (RSS): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-service-plan-v2.json.tmp file = common/src/ledger.rs:198
2722023-12-06T05:04:06.815ZINFOSledAgent (RSS): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-service-plan-v2.json.tmp file = common/src/ledger.rs:198
2732023-12-06T05:04:06.815ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:749
2742023-12-06T05:04:06.847ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 351d6c62-1449-4a75-aaf7-2115096c3894, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
2752023-12-06T05:04:06.847ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40843
2762023-12-06T05:04:06.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
2772023-12-06T05:04:06.847ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
2782023-12-06T05:04:06.889ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
2792023-12-06T05:04:06.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
2802023-12-06T05:04:06.898ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
2812023-12-06T05:04:06.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
2822023-12-06T05:04:06.946ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
2832023-12-06T05:04:06.990ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2842023-12-06T05:04:06.990ZWARNSledAgent (ServiceManager): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:187
2852023-12-06T05:04:06.990ZERROSledAgent (ServiceManager): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json
2862023-12-06T05:04:06.990ZWARNSledAgent (ServiceManager): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:187
2872023-12-06T05:04:06.990ZERROSledAgent (ServiceManager): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json
2882023-12-06T05:04:06.990ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2892023-12-06T05:04:06.993ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2902023-12-06T05:04:06.994ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
2912023-12-06T05:04:06.994ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2922023-12-06T05:04:06.996ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2932023-12-06T05:04:07.050ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894 file = illumos-utils/src/zone.rs:314
2942023-12-06T05:04:07.054ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c file = illumos-utils/src/zone.rs:314
2952023-12-06T05:04:07.058ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35 file = illumos-utils/src/zone.rs:314
2962023-12-06T05:04:07.070ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894 file = illumos-utils/src/zone.rs:353
2972023-12-06T05:04:07.075ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c file = illumos-utils/src/zone.rs:353
2982023-12-06T05:04:07.080ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35 file = illumos-utils/src/zone.rs:353
2992023-12-06T05:04:07.093ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 734.502978ms sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3002023-12-06T05:04:07.130ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3012023-12-06T05:04:07.261ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3022023-12-06T05:04:07.491ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3032023-12-06T05:04:07.745ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3042023-12-06T05:04:07.829ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 1.352258053s sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3052023-12-06T05:04:07.967ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3062023-12-06T05:04:08.482ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3072023-12-06T05:04:08.739ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3082023-12-06T05:04:08.777ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3092023-12-06T05:04:08.785ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3102023-12-06T05:04:08.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 36.596714ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3112023-12-06T05:04:08.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 43.780501ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3122023-12-06T05:04:08.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 46.396629ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3132023-12-06T05:04:08.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 116.191123ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3142023-12-06T05:04:09.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 92.429591ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3152023-12-06T05:04:09.043ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3162023-12-06T05:04:09.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 114.838977ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3172023-12-06T05:04:09.224ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
3182023-12-06T05:04:09.224ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 1.126191886s sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3192023-12-06T05:04:09.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 143.117924ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3202023-12-06T05:04:09.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 289.294488ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3212023-12-06T05:04:09.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 243.960326ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3222023-12-06T05:04:09.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 311.38456ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3232023-12-06T05:04:09.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 280.752146ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3242023-12-06T05:04:09.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 224.389516ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3252023-12-06T05:04:09.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 660.869785ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3262023-12-06T05:04:09.787ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3272023-12-06T05:04:09.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 736.909819ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3282023-12-06T05:04:09.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 445.164582ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3292023-12-06T05:04:10.269ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:a02a::2 exists in switch zone file = sled-agent/src/services.rs:1733
3302023-12-06T05:04:10.269ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:625 zone = oxz_switch
3312023-12-06T05:04:10.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 1.31962081s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3322023-12-06T05:04:10.312ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 4.262823379s sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3332023-12-06T05:04:10.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 1.198641444s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3342023-12-06T05:04:10.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 1.353574181s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3352023-12-06T05:04:10.597ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3362023-12-06T05:04:11.153ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3372023-12-06T05:04:11.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 1.158814135s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3382023-12-06T05:04:11.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 613.271721ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3392023-12-06T05:04:11.819ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3402023-12-06T05:04:11.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 1.201917604s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3412023-12-06T05:04:12.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 861.53099ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3422023-12-06T05:04:12.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 769.249905ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3432023-12-06T05:04:12.968ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3442023-12-06T05:04:13.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894") failed: Property not found. retry in 731.209422ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3452023-12-06T05:04:13.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35") failed: Property not found. retry in 1.187574626s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3462023-12-06T05:04:13.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c") failed: Property not found. retry in 900.735942ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3472023-12-06T05:04:13.908ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:2::1 exists file = sled-agent/src/services.rs:1769
3482023-12-06T05:04:13.908ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:2::1, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_internal_dns_351d6c62-1449-4a75-aaf7-2115096c3894
3492023-12-06T05:04:14.474ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:2::1 exists - OK file = sled-agent/src/services.rs:1777
3502023-12-06T05:04:14.474ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3512023-12-06T05:04:14.640ZINFOSledAgent (ServiceManager): Setting up internal-dns service file = sled-agent/src/services.rs:1995
3522023-12-06T05:04:14.802ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fece:d753 file = sled-agent/src/services.rs:1740
3532023-12-06T05:04:14.808ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3542023-12-06T05:04:14.927ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2280
3552023-12-06T05:04:15.171ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
3562023-12-06T05:04:15.171ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095 sled_id = 81c25e4c-3681-431d-a398-2d4416a62095
3572023-12-06T05:04:15.175ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
3582023-12-06T05:04:15.175ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2159
3592023-12-06T05:04:15.227ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:1::1 exists file = sled-agent/src/services.rs:1769
3602023-12-06T05:04:15.228ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:1::1, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c
3612023-12-06T05:04:15.483ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2194
3622023-12-06T05:04:15.680ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2461
3632023-12-06T05:04:15.778ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:1::1 exists - OK file = sled-agent/src/services.rs:1777
3642023-12-06T05:04:15.778ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3652023-12-06T05:04:15.824ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2466
3662023-12-06T05:04:15.952ZINFOSledAgent (ServiceManager): Setting up internal-dns service file = sled-agent/src/services.rs:1995
3672023-12-06T05:04:16.005ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2400
3682023-12-06T05:04:16.170ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2188
3692023-12-06T05:04:16.180ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3294 new = [fd00:1122:3344:101::2, ::1] old = [::1]
3702023-12-06T05:04:16.180ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3310
3712023-12-06T05:04:16.180ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_switch
3722023-12-06T05:04:16.505ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:3::1 exists file = sled-agent/src/services.rs:1769
3732023-12-06T05:04:16.505ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:3::1, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_internal_dns_2587e5bc-9a30-40ee-9e8c-0df00b89fe35
3742023-12-06T05:04:16.735ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3318
3752023-12-06T05:04:16.742ZINFOSledAgent (ServiceManager): configuring dendrite zone file = sled-agent/src/services.rs:3374
3762023-12-06T05:04:16.928ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3414 rack_subnet = fd00:1122:3344::
3772023-12-06T05:04:16.981ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:359
3782023-12-06T05:04:17.013ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 38.871117ms) error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "d896b6fe-e1d7-4972-a55b-343420db00f7", "content-length": "152", "date": "Wed, 06 Dec 2023 05:04:16 GMT"}; value: Error { error_code: Some("DiscoveryNotYetComplete"), message: "discovery process not yet complete", request_id: "d896b6fe-e1d7-4972-a55b-343420db00f7" } file = sled-agent/src/bootstrap/early_networking.rs:378
3792023-12-06T05:04:17.052ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:3::1 exists - OK file = sled-agent/src/services.rs:1777
3802023-12-06T05:04:17.052ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3812023-12-06T05:04:17.056ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 134.483718ms) error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "7a3b7278-5714-428d-ac1c-3816ce509af5", "content-length": "152", "date": "Wed, 06 Dec 2023 05:04:16 GMT"}; value: Error { error_code: Some("DiscoveryNotYetComplete"), message: "discovery process not yet complete", request_id: "7a3b7278-5714-428d-ac1c-3816ce509af5" } file = sled-agent/src/bootstrap/early_networking.rs:378
3822023-12-06T05:04:17.192ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 109.517055ms) error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "00d0fc08-906e-48d0-8de6-0f335b798935", "content-length": "152", "date": "Wed, 06 Dec 2023 05:04:16 GMT"}; value: Error { error_code: Some("DiscoveryNotYetComplete"), message: "discovery process not yet complete", request_id: "00d0fc08-906e-48d0-8de6-0f335b798935" } file = sled-agent/src/bootstrap/early_networking.rs:378
3832023-12-06T05:04:17.229ZINFOSledAgent (ServiceManager): Setting up internal-dns service file = sled-agent/src/services.rs:1995
3842023-12-06T05:04:17.303ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:407
3852023-12-06T05:04:17.336ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:533
3862023-12-06T05:04:17.336ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:602
3872023-12-06T05:04:17.337ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:605 uptime = 2
3882023-12-06T05:04:17.337ZINFOSledAgent (ServiceManager): Configuring boundary services loopback address on switch config = Ipv6Entry {\n addr: fd00:99::1,\n tag: "omicron",\n} file = sled-agent/src/bootstrap/early_networking.rs:429
3892023-12-06T05:04:17.337ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n v4_routes: {\n "0.0.0.0/0": [\n RouteSettingsV4 {\n link_id: 0,\n nexthop: 10.151.2.100,\n },\n ],\n },\n v6_routes: {},\n} file = sled-agent/src/bootstrap/early_networking.rs:440
3902023-12-06T05:04:17.338ZINFOSledAgent (ServiceManager): advertising boundary services loopback address file = sled-agent/src/bootstrap/early_networking.rs:457
3912023-12-06T05:04:17.404ZINFOSledAgent (ServiceManager): Sending prefix to ddmd for advertisement DdmAdminClient = [fd00:1122:3344:101::2]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:99::, len: 64 }
3922023-12-06T05:04:17.448ZINFOSledAgent (StorageManager): Received DisksChanged({}) file = sled-storage/src/manager.rs:346
3932023-12-06T05:04:17.708ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
3942023-12-06T05:04:17.760ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
3952023-12-06T05:04:17.760ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
3962023-12-06T05:04:17.760ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
3972023-12-06T05:04:17.760ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 10913567 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:40843 req_id = 18132e7e-9258-4b56-8859-aaa9f1510e3a response_code = 204 uri = /omicron-zones
3982023-12-06T05:04:17.762ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:417
3992023-12-06T05:04:17.802ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:417
4002023-12-06T05:04:17.848ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:417
4012023-12-06T05:04:19.059ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:453
4022023-12-06T05:04:19.059ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
4032023-12-06T05:04:19.091ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
4042023-12-06T05:04:19.092ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
4052023-12-06T05:04:19.092ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
4062023-12-06T05:04:19.121ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 351d6c62-1449-4a75-aaf7-2115096c3894, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: ac41f50b-a9b5-4db0-979b-f03899fc89ef, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: df92de65-bfac-4217-9171-69e0dacb3b06, kind: Service { id: ac41f50b-a9b5-4db0-979b-f03899fc89ef }, name: Name("ntp-ac41f50b-a9b5-4db0-979b-f03899fc89ef"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 13])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
4072023-12-06T05:04:19.121ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48204
4082023-12-06T05:04:19.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
4092023-12-06T05:04:19.121ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
4102023-12-06T05:04:19.142ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
4112023-12-06T05:04:19.142ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
4122023-12-06T05:04:19.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
4132023-12-06T05:04:19.164ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
4142023-12-06T05:04:19.183ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
4152023-12-06T05:04:19.183ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
4162023-12-06T05:04:19.184ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
4172023-12-06T05:04:19.184ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
4182023-12-06T05:04:19.186ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
4192023-12-06T05:04:19.218ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
4202023-12-06T05:04:19.219ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
4212023-12-06T05:04:19.219ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
4222023-12-06T05:04:19.264ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte0", _ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 177, 13]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
4232023-12-06T05:04:19.264ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = ntp
4242023-12-06T05:04:19.265ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
4252023-12-06T05:04:19.265ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
4262023-12-06T05:04:19.265ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
4272023-12-06T05:04:19.266ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
4282023-12-06T05:04:19.293ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef file = illumos-utils/src/zone.rs:314
4292023-12-06T05:04:19.312ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef file = illumos-utils/src/zone.rs:353
4302023-12-06T05:04:20.176ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
4312023-12-06T05:04:20.309ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4322023-12-06T05:04:20.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 34.76884ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4332023-12-06T05:04:20.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 96.765695ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4342023-12-06T05:04:20.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 155.538681ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4352023-12-06T05:04:20.769ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52493
4362023-12-06T05:04:20.769ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 96 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52493 req_id = 4b799969-e50e-4511-9caf-20abf356974f response_code = 200 uri = /baseboard
4372023-12-06T05:04:20.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 472.472265ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4382023-12-06T05:04:21.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 801.715338ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4392023-12-06T05:04:22.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 1.160090437s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4402023-12-06T05:04:23.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 1.368712519s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4412023-12-06T05:04:24.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef") failed: Property not found. retry in 1.337167711s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4422023-12-06T05:04:25.177ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
4432023-12-06T05:04:26.011ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::1b exists file = sled-agent/src/services.rs:1769
4442023-12-06T05:04:26.011ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::1b, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4452023-12-06T05:04:26.562ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::1b exists - OK file = sled-agent/src/services.rs:1777
4462023-12-06T05:04:26.562ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
4472023-12-06T05:04:26.759ZINFOSledAgent (ServiceManager): Set up NTP service boundary=true, Servers=["0.pool.ntp.org"] file = sled-agent/src/services.rs:2086
4482023-12-06T05:04:26.785ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef
4492023-12-06T05:04:31.818ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
4502023-12-06T05:04:31.866ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
4512023-12-06T05:04:31.866ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
4522023-12-06T05:04:31.867ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 12745731 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48204 req_id = b3c3d69a-c6bf-4640-9585-d63a7083b7c7 response_code = 204 uri = /omicron-zones
4532023-12-06T05:04:31.869ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:494
4542023-12-06T05:04:31.899ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4552023-12-06T05:04:31.899ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63976
4562023-12-06T05:04:31.910ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_ac41f50b-a9b5-4db0-979b-f03899fc89ef': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3028
4572023-12-06T05:04:31.910ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 10281 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63976 req_id = f56235c2-45a5-4192-a08b-9cb77eca7a41 response_code = 500 uri = /timesync
4582023-12-06T05:04:31.913ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4592023-12-06T05:04:32.021ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4602023-12-06T05:04:32.021ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59815
4612023-12-06T05:04:32.030ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8928 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59815 req_id = fd020ee7-03b0-4284-8ba8-e10e2fd3d273 response_code = 200 uri = /timesync
4622023-12-06T05:04:32.033ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4632023-12-06T05:04:32.033ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4642023-12-06T05:04:32.190ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4652023-12-06T05:04:32.190ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51008
4662023-12-06T05:04:32.199ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 9031 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51008 req_id = d85da47b-d82a-47d9-8003-1f9bcbc52b9d response_code = 200 uri = /timesync
4672023-12-06T05:04:32.202ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4682023-12-06T05:04:32.202ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4692023-12-06T05:04:32.448ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4702023-12-06T05:04:32.448ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35052
4712023-12-06T05:04:32.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8037 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35052 req_id = c2005462-d270-4f5f-b411-58e343fdb2d3 response_code = 200 uri = /timesync
4722023-12-06T05:04:32.458ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4732023-12-06T05:04:32.458ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4742023-12-06T05:04:32.669ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4752023-12-06T05:04:32.669ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42814
4762023-12-06T05:04:32.678ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8068 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42814 req_id = 54e6ce86-9dd4-4648-af74-ff1d5e96fe04 response_code = 200 uri = /timesync
4772023-12-06T05:04:32.680ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4782023-12-06T05:04:32.680ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4792023-12-06T05:04:32.836ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4802023-12-06T05:04:32.836ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56252
4812023-12-06T05:04:32.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 9021 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56252 req_id = b4922712-0e96-43b1-813a-df93173b1fb6 response_code = 200 uri = /timesync
4822023-12-06T05:04:32.848ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4832023-12-06T05:04:32.848ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4842023-12-06T05:04:33.210ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4852023-12-06T05:04:33.210ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35172
4862023-12-06T05:04:33.218ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8043 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35172 req_id = e1716557-8493-4513-a6a1-f1eeff1514b7 response_code = 200 uri = /timesync
4872023-12-06T05:04:33.221ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4882023-12-06T05:04:33.221ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4892023-12-06T05:04:33.637ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4902023-12-06T05:04:33.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57568
4912023-12-06T05:04:33.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 9448 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57568 req_id = 20f40cee-faef-41b9-b3df-bec58d3a05a5 response_code = 200 uri = /timesync
4922023-12-06T05:04:33.649ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4932023-12-06T05:04:33.649ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4942023-12-06T05:04:34.189ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4952023-12-06T05:04:34.189ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34901
4962023-12-06T05:04:34.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8258 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34901 req_id = 63b823b8-1bf4-44d5-a428-58a9c5998b32 response_code = 200 uri = /timesync
4972023-12-06T05:04:34.199ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4982023-12-06T05:04:34.199ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4992023-12-06T05:04:34.687ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5002023-12-06T05:04:34.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37862
5012023-12-06T05:04:34.695ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7785 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37862 req_id = 87aa37bc-855b-4858-8fa8-d366c3d0b6a3 response_code = 200 uri = /timesync
5022023-12-06T05:04:34.697ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5032023-12-06T05:04:34.697ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5042023-12-06T05:04:35.129ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5052023-12-06T05:04:35.129ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58882
5062023-12-06T05:04:35.137ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7877 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58882 req_id = d29bb72a-8014-4f72-b868-63bc567fa53f response_code = 200 uri = /timesync
5072023-12-06T05:04:35.139ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5082023-12-06T05:04:35.139ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5092023-12-06T05:04:35.838ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5102023-12-06T05:04:35.838ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43251
5112023-12-06T05:04:35.847ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8641 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43251 req_id = c60c7958-75da-4209-a9ac-6cd4b67350eb response_code = 200 uri = /timesync
5122023-12-06T05:04:35.849ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5132023-12-06T05:04:35.849ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5142023-12-06T05:04:36.547ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5152023-12-06T05:04:36.547ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52729
5162023-12-06T05:04:36.556ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8673 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52729 req_id = 3118f5d2-efb5-445c-83ac-b5505706a26f response_code = 200 uri = /timesync
5172023-12-06T05:04:36.558ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5182023-12-06T05:04:36.558ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5192023-12-06T05:04:36.822ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
5202023-12-06T05:04:37.057ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5212023-12-06T05:04:37.057ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42182
5222023-12-06T05:04:37.064ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7380 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42182 req_id = ac803a0f-ee9e-40a3-8661-1050b2f19cd3 response_code = 200 uri = /timesync
5232023-12-06T05:04:37.068ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5242023-12-06T05:04:37.068ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5252023-12-06T05:04:38.136ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5262023-12-06T05:04:38.136ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64594
5272023-12-06T05:04:38.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 9142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64594 req_id = fa0335d4-76cb-448a-b453-18f224bc1d15 response_code = 200 uri = /timesync
5282023-12-06T05:04:38.148ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701839070.9179864, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5292023-12-06T05:04:38.148ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5302023-12-06T05:04:38.705ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5312023-12-06T05:04:38.705ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37908
5322023-12-06T05:04:38.710ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
5332023-12-06T05:04:38.725ZWARNSledAgent (ServiceManager): Updating [wu]tmpx databases failed: Command [/usr/platform/oxide/bin/tmpx -Z] executed and failed with status: exit status: 1 stdout: [zone:global:0] Boot time: 1701838500 Root path: Updating database '/var/adm/wtmpx' First entry looks correct; nothing to do Updating database '/var/adm/utmpx' First entry looks correct; nothing to do [zone:sidecar_softnpu:1] Boot time: 1701839027 Root path: /sidecar/sidecar_softnpu/root Updating database '/sidecar/sidecar_softnpu/root/var/adm/wtmpx' First entry looks correct; nothing to do Updating database '/sidecar/sidecar_softnpu/root/var/adm/utmpx' First entry looks correct; nothing to do [zone:oxz_switch:2] Boot time: 1701839039 Root path: /zone/oxz_switch/root Updating database '/zone/oxz_switch/root/var/adm/wtmpx' First entry looks correct; nothing to do Updating database '/zone/oxz_switch/root/var/adm/utmpx' First entry looks correct; nothing to do [zone:oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c:3] Boot time: 1701839048 Root path: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c/root Updating database '/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c/root/var/adm/wtmpx' stderr: tmpx: Invalid database: '/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_64e16771-3f1e-4d2a-99c2-117bd5fe2a0c/root/var/adm/wtmpx' (see utmpxname(3C)) file = sled-agent/src/services.rs:2949
5342023-12-06T05:04:38.725ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 19524 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37908 req_id = e41b18db-0226-4c64-97a7-16ebcd6fb1da response_code = 200 uri = /timesync
5352023-12-06T05:04:38.727ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 89016161, ip_addr: 5.78.71.97, stratum: 3, ref_time: 1701839076.1561482, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5362023-12-06T05:04:38.727ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1042
5372023-12-06T05:04:38.756ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 351d6c62-1449-4a75-aaf7-2115096c3894, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: ac41f50b-a9b5-4db0-979b-f03899fc89ef, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: df92de65-bfac-4217-9171-69e0dacb3b06, kind: Service { id: ac41f50b-a9b5-4db0-979b-f03899fc89ef }, name: Name("ntp-ac41f50b-a9b5-4db0-979b-f03899fc89ef"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 13])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2b7857a9-72a7-4e49-84f1-64a642107b35, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 49e1e0a4-7ede-41b5-888a-00b868df57eb, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 1699f35f-125f-4f89-b1cb-ddb6e119621a, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } } } }, OmicronZoneConfig { id: 6a88e90d-fac6-4cc4-ab36-7c7a4124aef1, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } } } }, OmicronZoneConfig { id: dff9a72d-176e-4b65-8106-0d1266f6203b, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
5382023-12-06T05:04:38.757ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49356
5392023-12-06T05:04:38.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5402023-12-06T05:04:38.757ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5412023-12-06T05:04:38.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5422023-12-06T05:04:38.777ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5432023-12-06T05:04:38.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5442023-12-06T05:04:38.797ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5452023-12-06T05:04:38.817ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2b7857a9-72a7-4e49-84f1-64a642107b35, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5462023-12-06T05:04:38.817ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 2b7857a9-72a7-4e49-84f1-64a642107b35, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5472023-12-06T05:04:38.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 49e1e0a4-7ede-41b5-888a-00b868df57eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5482023-12-06T05:04:38.867ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 49e1e0a4-7ede-41b5-888a-00b868df57eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5492023-12-06T05:04:38.918ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1699f35f-125f-4f89-b1cb-ddb6e119621a, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5502023-12-06T05:04:38.918ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 1699f35f-125f-4f89-b1cb-ddb6e119621a, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5512023-12-06T05:04:38.966ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a88e90d-fac6-4cc4-ab36-7c7a4124aef1, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5522023-12-06T05:04:38.966ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 6a88e90d-fac6-4cc4-ab36-7c7a4124aef1, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5532023-12-06T05:04:39.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
5542023-12-06T05:04:39.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dff9a72d-176e-4b65-8106-0d1266f6203b, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5552023-12-06T05:04:39.024ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: dff9a72d-176e-4b65-8106-0d1266f6203b, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5562023-12-06T05:04:39.073ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5572023-12-06T05:04:39.073ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
5582023-12-06T05:04:39.073ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
5592023-12-06T05:04:39.073ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5602023-12-06T05:04:39.080ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5612023-12-06T05:04:39.080ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5622023-12-06T05:04:39.080ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5632023-12-06T05:04:39.080ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5642023-12-06T05:04:39.080ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5652023-12-06T05:04:39.172ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1 file = illumos-utils/src/zone.rs:314
5662023-12-06T05:04:39.176ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb file = illumos-utils/src/zone.rs:314
5672023-12-06T05:04:39.180ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b file = illumos-utils/src/zone.rs:314
5682023-12-06T05:04:39.185ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a file = illumos-utils/src/zone.rs:314
5692023-12-06T05:04:39.189ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35 file = illumos-utils/src/zone.rs:314
5702023-12-06T05:04:39.193ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1 file = illumos-utils/src/zone.rs:353
5712023-12-06T05:04:39.198ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb file = illumos-utils/src/zone.rs:353
5722023-12-06T05:04:39.202ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b file = illumos-utils/src/zone.rs:353
5732023-12-06T05:04:39.207ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a file = illumos-utils/src/zone.rs:353
5742023-12-06T05:04:39.211ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35 file = illumos-utils/src/zone.rs:353
5752023-12-06T05:04:41.318ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
5762023-12-06T05:04:42.718ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::4"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5772023-12-06T05:04:42.718ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
5782023-12-06T05:04:42.726ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::7"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5792023-12-06T05:04:42.726ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
5802023-12-06T05:04:42.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 29.17952ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
5812023-12-06T05:04:42.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 26.694402ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
5822023-12-06T05:04:42.882ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::6"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5832023-12-06T05:04:42.882ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
5842023-12-06T05:04:42.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 85.565742ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
5852023-12-06T05:04:42.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 78.620967ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
5862023-12-06T05:04:43.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 104.326495ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
5872023-12-06T05:04:43.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 215.270847ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
5882023-12-06T05:04:43.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 574.529967ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
5892023-12-06T05:04:43.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 510.335516ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
5902023-12-06T05:04:43.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 51.063052ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
5912023-12-06T05:04:43.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 142.669793ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
5922023-12-06T05:04:43.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 258.800823ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
5932023-12-06T05:04:43.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 486.916056ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
5942023-12-06T05:04:43.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 740.678643ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
5952023-12-06T05:04:43.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 576.612227ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
5962023-12-06T05:04:44.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 943.390305ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
5972023-12-06T05:04:44.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 1.033270471s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
5982023-12-06T05:04:44.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 1.145776854s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
5992023-12-06T05:04:44.694ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::5"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6002023-12-06T05:04:44.694ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6012023-12-06T05:04:44.757ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::3"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6022023-12-06T05:04:44.758ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6032023-12-06T05:04:44.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 58.294802ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6042023-12-06T05:04:44.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 79.618165ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6052023-12-06T05:04:44.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 68.402491ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6062023-12-06T05:04:44.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 124.820938ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6072023-12-06T05:04:44.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 130.923352ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6082023-12-06T05:04:45.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 390.762349ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6092023-12-06T05:04:45.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 232.35029ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6102023-12-06T05:04:45.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 897.518525ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6112023-12-06T05:04:45.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 209.739218ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6122023-12-06T05:04:45.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 557.612017ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6132023-12-06T05:04:45.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 939.184597ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
6142023-12-06T05:04:45.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 1.008419088s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6152023-12-06T05:04:45.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 766.984014ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
6162023-12-06T05:04:46.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 553.806454ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6172023-12-06T05:04:46.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 1.265872906s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6182023-12-06T05:04:46.319ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6192023-12-06T05:04:46.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 1.361258298s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
6202023-12-06T05:04:46.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 1.366395935s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
6212023-12-06T05:04:46.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 564.853149ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6222023-12-06T05:04:46.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 1.262088171s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6232023-12-06T05:04:47.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 564.615346ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6242023-12-06T05:04:47.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 651.806517ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6252023-12-06T05:04:47.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 830.020295ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6262023-12-06T05:04:47.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 824.782879ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
6272023-12-06T05:04:47.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 1.226002673s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6282023-12-06T05:04:47.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 1.371478691s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
6292023-12-06T05:04:48.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 1.393548896s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6302023-12-06T05:04:48.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 1.201077446s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6312023-12-06T05:04:48.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 1.095081636s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
6322023-12-06T05:04:49.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 1.243857673s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6332023-12-06T05:04:49.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 1.107764481s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
6342023-12-06T05:04:49.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 1.02339798s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6352023-12-06T05:04:49.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 527.4682ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6362023-12-06T05:04:49.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 1.284814222s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
6372023-12-06T05:04:50.296ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45611
6382023-12-06T05:04:50.296ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 68 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45611 req_id = 23b8fa8d-4c8d-43d2-a2f6-b553fd7ea546 response_code = 200 uri = /baseboard
6392023-12-06T05:04:50.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 926.192099ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6402023-12-06T05:04:50.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 1.494142888s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6412023-12-06T05:04:50.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 1.034385614s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
6422023-12-06T05:04:50.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 593.517462ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6432023-12-06T05:04:51.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 854.21342ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
6442023-12-06T05:04:51.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 1.020724945s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6452023-12-06T05:04:51.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 1.447556568s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6462023-12-06T05:04:51.321ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6472023-12-06T05:04:51.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb") failed: Property not found. retry in 1.324095713s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_49e1e0a4-7ede-41b5-888a-00b868df57eb
6482023-12-06T05:04:51.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a") failed: Property not found. retry in 1.181551203s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a
6492023-12-06T05:04:51.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1") failed: Property not found. retry in 1.460458875s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a88e90d-fac6-4cc4-ab36-7c7a4124aef1
6502023-12-06T05:04:52.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b") failed: Property not found. retry in 772.763582ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dff9a72d-176e-4b65-8106-0d1266f6203b
6512023-12-06T05:04:52.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35") failed: Property not found. retry in 1.080926282s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2b7857a9-72a7-4e49-84f1-64a642107b35
6522023-12-06T05:04:53.858ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
6532023-12-06T05:04:53.858ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
6542023-12-06T05:04:53.859ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 15092943 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49356 req_id = f7f2e1f9-0c99-4fc9-ba80-5ed546b0f60d response_code = 204 uri = /omicron-zones
6552023-12-06T05:04:53.903ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37570
6562023-12-06T05:04:53.935ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 32149 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = 9f4dbc66-be79-4b97-973e-536da489b6ce response_code = 500 uri = /cockroachdb
6572023-12-06T05:04:53.936ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9f4dbc66-be79-4b97-973e-536da489b6ce", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9f4dbc66-be79-4b97-973e-536da489b6ce" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 94.268041ms
6582023-12-06T05:04:54.057ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 26189 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = 43518ad7-67cb-4bb9-a53d-bf015fc99202 response_code = 500 uri = /cockroachdb
6592023-12-06T05:04:54.057ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "43518ad7-67cb-4bb9-a53d-bf015fc99202", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "43518ad7-67cb-4bb9-a53d-bf015fc99202" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 91.721934ms
6602023-12-06T05:04:54.177ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 27745 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = d111a911-f67c-4237-9feb-612c331af699 response_code = 500 uri = /cockroachdb
6612023-12-06T05:04:54.177ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d111a911-f67c-4237-9feb-612c331af699", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d111a911-f67c-4237-9feb-612c331af699" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 196.663086ms
6622023-12-06T05:04:54.402ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 26998 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = cd76fa91-62fc-4848-b666-02311e48a86d response_code = 500 uri = /cockroachdb
6632023-12-06T05:04:54.402ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cd76fa91-62fc-4848-b666-02311e48a86d", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cd76fa91-62fc-4848-b666-02311e48a86d" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 181.059519ms
6642023-12-06T05:04:54.611ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 26725 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = 8cecb4cb-c5af-42a3-9b96-ee0c625e7449 response_code = 500 uri = /cockroachdb
6652023-12-06T05:04:54.611ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8cecb4cb-c5af-42a3-9b96-ee0c625e7449", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8cecb4cb-c5af-42a3-9b96-ee0c625e7449" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 200.168383ms
6662023-12-06T05:04:54.839ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 25283 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = b45dd93d-d953-44ef-9695-a8efb4ef4730 response_code = 500 uri = /cockroachdb
6672023-12-06T05:04:54.839ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b45dd93d-d953-44ef-9695-a8efb4ef4730", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b45dd93d-d953-44ef-9695-a8efb4ef4730" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 199.025751ms
6682023-12-06T05:04:55.066ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 27454 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = 4e0da932-ed4f-47bd-9121-99b801b08824 response_code = 500 uri = /cockroachdb
6692023-12-06T05:04:55.067ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4e0da932-ed4f-47bd-9121-99b801b08824", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4e0da932-ed4f-47bd-9121-99b801b08824" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 327.788716ms
6702023-12-06T05:04:55.426ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 29638 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = 73a0ebf0-f092-43d9-987b-c0eb656bc52a response_code = 500 uri = /cockroachdb
6712023-12-06T05:04:55.426ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "73a0ebf0-f092-43d9-987b-c0eb656bc52a", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "73a0ebf0-f092-43d9-987b-c0eb656bc52a" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 421.856441ms
6722023-12-06T05:04:55.876ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService8/omicron6 in oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1699f35f-125f-4f89-b1cb-ddb6e119621a /usr/sbin/ipadm show-addr -p -o ADDR oxControlService8/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 27084 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = b646fd7d-5039-4083-93c2-bea5fee28bc4 response_code = 500 uri = /cockroachdb
6732023-12-06T05:04:55.876ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b646fd7d-5039-4083-93c2-bea5fee28bc4", "content-length": "124", "date": "Wed, 06 Dec 2023 05:04:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b646fd7d-5039-4083-93c2-bea5fee28bc4" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 501.645257ms
6742023-12-06T05:04:56.322ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6752023-12-06T05:04:56.411ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:2881
6762023-12-06T05:05:04.281ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:2899
6772023-12-06T05:05:06.788ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
6782023-12-06T05:05:15.185ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:2920
6792023-12-06T05:05:15.185ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 18493333 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37570 req_id = a4024657-307b-49df-98e3-fedeb8ee282a response_code = 204 uri = /cockroachdb
6802023-12-06T05:05:15.186ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6812023-12-06T05:05:15.220ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 351d6c62-1449-4a75-aaf7-2115096c3894, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: ac41f50b-a9b5-4db0-979b-f03899fc89ef, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: df92de65-bfac-4217-9171-69e0dacb3b06, kind: Service { id: ac41f50b-a9b5-4db0-979b-f03899fc89ef }, name: Name("ntp-ac41f50b-a9b5-4db0-979b-f03899fc89ef"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 13])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2b7857a9-72a7-4e49-84f1-64a642107b35, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 49e1e0a4-7ede-41b5-888a-00b868df57eb, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 1699f35f-125f-4f89-b1cb-ddb6e119621a, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } } } }, OmicronZoneConfig { id: 6a88e90d-fac6-4cc4-ab36-7c7a4124aef1, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } } } }, OmicronZoneConfig { id: dff9a72d-176e-4b65-8106-0d1266f6203b, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } } } }, OmicronZoneConfig { id: fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 5f7538d9-7dd9-4257-a4dd-751c9f1af140, kind: Service { id: fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4 }, name: Name("external-dns-fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 106])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 72cbd99a-a71b-4108-ac9e-b67ccdec1db9, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 13c5ab6d-2c8e-458a-bd69-abf74fb65618, kind: Service { id: 72cbd99a-a71b-4108-ac9e-b67ccdec1db9 }, name: Name("external-dns-72cbd99a-a71b-4108-ac9e-b67ccdec1db9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 34])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 8dde7bf4-95ce-4d31-8b97-4393ef169a80, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: b364d85f-e703-447f-98b5-e0cc6ff589d3, kind: Service { id: 8dde7bf4-95ce-4d31-8b97-4393ef169a80 }, name: Name("nexus-8dde7bf4-95ce-4d31-8b97-4393ef169a80"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 172, 159])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: e1cbca33-5880-4c04-84b8-c054468b5065, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 0d17d4ea-9aa2-43b9-8a3c-7f7650f77c7d, kind: Service { id: e1cbca33-5880-4c04-84b8-c054468b5065 }, name: Name("nexus-e1cbca33-5880-4c04-84b8-c054468b5065"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 120])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 983419ea-e2f5-4b1a-ac1a-57f106744087, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: cb9210f0-f677-49dd-8af4-6c86b07d954d, kind: Service { id: 983419ea-e2f5-4b1a-ac1a-57f106744087 }, name: Name("nexus-983419ea-e2f5-4b1a-ac1a-57f106744087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 199])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 46c23f11-8078-4578-83a9-a198e53b593d, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 646423a4-0018-4fb0-818f-0200cf3d138b, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 4eb02bef-3664-4549-bd92-a35a697fa51a, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 2f0db282-de17-4fdb-8606-247b3cd829a4, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a7e7761d-0fdf-4d42-a6b6-e5422167aa5e, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 8292c6ef-97da-43c9-94ae-c6763fefcb25, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 45fe60c0-65a2-4837-acb9-f15c15dda19a, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } } } }, OmicronZoneConfig { id: e4412f1c-f03e-41c8-b0ef-e638ffd17389, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } } } }, OmicronZoneConfig { id: 41877aea-1cd2-4d10-828c-6dc0a1b00f2d, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } } } }, OmicronZoneConfig { id: a658ba43-3f8c-4198-9a76-2c6392304e35, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External } } } }, OmicronZoneConfig { id: 5099d9bc-e314-42f3-a775-0218f54ecf4a, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External } } } }, OmicronZoneConfig { id: abbb3b13-4924-4c3e-93f6-083c065967fa, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: a31161d5-0c9f-4788-bd3f-3888f075c4b0, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
6822023-12-06T05:05:15.220ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64164
6832023-12-06T05:05:15.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6842023-12-06T05:05:15.221ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 64e16771-3f1e-4d2a-99c2-117bd5fe2a0c, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6852023-12-06T05:05:15.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6862023-12-06T05:05:15.244ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 351d6c62-1449-4a75-aaf7-2115096c3894, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6872023-12-06T05:05:15.266ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6882023-12-06T05:05:15.266ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 2587e5bc-9a30-40ee-9e8c-0df00b89fe35, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6892023-12-06T05:05:15.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2b7857a9-72a7-4e49-84f1-64a642107b35, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6902023-12-06T05:05:15.288ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 2b7857a9-72a7-4e49-84f1-64a642107b35, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6912023-12-06T05:05:15.308ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 49e1e0a4-7ede-41b5-888a-00b868df57eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6922023-12-06T05:05:15.308ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 49e1e0a4-7ede-41b5-888a-00b868df57eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6932023-12-06T05:05:15.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1699f35f-125f-4f89-b1cb-ddb6e119621a, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6942023-12-06T05:05:15.328ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 1699f35f-125f-4f89-b1cb-ddb6e119621a, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6952023-12-06T05:05:15.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a88e90d-fac6-4cc4-ab36-7c7a4124aef1, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6962023-12-06T05:05:15.349ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 6a88e90d-fac6-4cc4-ab36-7c7a4124aef1, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6972023-12-06T05:05:15.370ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dff9a72d-176e-4b65-8106-0d1266f6203b, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6982023-12-06T05:05:15.370ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: dff9a72d-176e-4b65-8106-0d1266f6203b, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6992023-12-06T05:05:15.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7002023-12-06T05:05:15.391ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7012023-12-06T05:05:15.479ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72cbd99a-a71b-4108-ac9e-b67ccdec1db9, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7022023-12-06T05:05:15.481ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 72cbd99a-a71b-4108-ac9e-b67ccdec1db9, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7032023-12-06T05:05:15.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 646423a4-0018-4fb0-818f-0200cf3d138b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Clickhouse }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7042023-12-06T05:05:15.554ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 646423a4-0018-4fb0-818f-0200cf3d138b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Clickhouse }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7052023-12-06T05:05:15.648ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7e7761d-0fdf-4d42-a6b6-e5422167aa5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7062023-12-06T05:05:15.648ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: a7e7761d-0fdf-4d42-a6b6-e5422167aa5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7072023-12-06T05:05:15.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8292c6ef-97da-43c9-94ae-c6763fefcb25, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7082023-12-06T05:05:15.722ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 8292c6ef-97da-43c9-94ae-c6763fefcb25, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7092023-12-06T05:05:15.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45fe60c0-65a2-4837-acb9-f15c15dda19a, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7102023-12-06T05:05:15.811ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 45fe60c0-65a2-4837-acb9-f15c15dda19a, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7112023-12-06T05:05:15.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e4412f1c-f03e-41c8-b0ef-e638ffd17389, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7122023-12-06T05:05:15.904ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: e4412f1c-f03e-41c8-b0ef-e638ffd17389, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7132023-12-06T05:05:15.988ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 41877aea-1cd2-4d10-828c-6dc0a1b00f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7142023-12-06T05:05:15.988ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 41877aea-1cd2-4d10-828c-6dc0a1b00f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7152023-12-06T05:05:16.082ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a658ba43-3f8c-4198-9a76-2c6392304e35, dataset_name: DatasetName { pool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7162023-12-06T05:05:16.082ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: a658ba43-3f8c-4198-9a76-2c6392304e35, dataset_name: DatasetName { pool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7172023-12-06T05:05:16.171ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5099d9bc-e314-42f3-a775-0218f54ecf4a, dataset_name: DatasetName { pool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7182023-12-06T05:05:16.171ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 5099d9bc-e314-42f3-a775-0218f54ecf4a, dataset_name: DatasetName { pool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7192023-12-06T05:05:16.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: abbb3b13-4924-4c3e-93f6-083c065967fa, dataset_name: DatasetName { pool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7202023-12-06T05:05:16.283ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: abbb3b13-4924-4c3e-93f6-083c065967fa, dataset_name: DatasetName { pool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7212023-12-06T05:05:16.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a31161d5-0c9f-4788-bd3f-3888f075c4b0, dataset_name: DatasetName { pool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7222023-12-06T05:05:16.376ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: a31161d5-0c9f-4788-bd3f-3888f075c4b0, dataset_name: DatasetName { pool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7232023-12-06T05:05:16.465ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7242023-12-06T05:05:16.465ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
7252023-12-06T05:05:16.465ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
7262023-12-06T05:05:16.465ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7272023-12-06T05:05:16.472ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7282023-12-06T05:05:16.472ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7292023-12-06T05:05:16.472ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7302023-12-06T05:05:16.472ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7312023-12-06T05:05:16.474ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7322023-12-06T05:05:16.485ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7332023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7342023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7352023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7362023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7372023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7382023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7392023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7402023-12-06T05:05:16.485ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7412023-12-06T05:05:16.487ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7422023-12-06T05:05:16.488ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7432023-12-06T05:05:16.491ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7442023-12-06T05:05:16.491ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7452023-12-06T05:05:16.491ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7462023-12-06T05:05:16.777ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a file = illumos-utils/src/zone.rs:314
7472023-12-06T05:05:16.782ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0 file = illumos-utils/src/zone.rs:314
7482023-12-06T05:05:16.787ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4 file = illumos-utils/src/zone.rs:314
7492023-12-06T05:05:16.792ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e file = illumos-utils/src/zone.rs:314
7502023-12-06T05:05:16.797ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d file = illumos-utils/src/zone.rs:314
7512023-12-06T05:05:16.802ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b file = illumos-utils/src/zone.rs:314
7522023-12-06T05:05:16.808ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d file = illumos-utils/src/zone.rs:314
7532023-12-06T05:05:16.813ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa file = illumos-utils/src/zone.rs:314
7542023-12-06T05:05:16.818ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389 file = illumos-utils/src/zone.rs:314
7552023-12-06T05:05:16.823ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35 file = illumos-utils/src/zone.rs:314
7562023-12-06T05:05:16.828ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a file = illumos-utils/src/zone.rs:353
7572023-12-06T05:05:16.833ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0 file = illumos-utils/src/zone.rs:353
7582023-12-06T05:05:16.837ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4 file = illumos-utils/src/zone.rs:353
7592023-12-06T05:05:16.843ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e file = illumos-utils/src/zone.rs:353
7602023-12-06T05:05:16.848ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d file = illumos-utils/src/zone.rs:353
7612023-12-06T05:05:16.852ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b file = illumos-utils/src/zone.rs:353
7622023-12-06T05:05:16.857ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d file = illumos-utils/src/zone.rs:353
7632023-12-06T05:05:16.862ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa file = illumos-utils/src/zone.rs:353
7642023-12-06T05:05:16.867ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389 file = illumos-utils/src/zone.rs:353
7652023-12-06T05:05:16.872ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35 file = illumos-utils/src/zone.rs:353
7662023-12-06T05:05:16.878ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a file = illumos-utils/src/zone.rs:314
7672023-12-06T05:05:16.883ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff file = illumos-utils/src/zone.rs:314
7682023-12-06T05:05:16.889ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25 file = illumos-utils/src/zone.rs:314
7692023-12-06T05:05:16.894ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a file = illumos-utils/src/zone.rs:314
7702023-12-06T05:05:16.904ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a file = illumos-utils/src/zone.rs:353
7712023-12-06T05:05:16.911ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff file = illumos-utils/src/zone.rs:353
7722023-12-06T05:05:16.918ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25 file = illumos-utils/src/zone.rs:353
7732023-12-06T05:05:16.924ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a file = illumos-utils/src/zone.rs:353
7742023-12-06T05:05:16.965ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7752023-12-06T05:05:17.001ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7762023-12-06T05:05:17.055ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7772023-12-06T05:05:17.095ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7782023-12-06T05:05:17.129ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7792023-12-06T05:05:17.131ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7802023-12-06T05:05:17.131ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7812023-12-06T05:05:17.201ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte1", _ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 172, 159]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" } }
7822023-12-06T05:05:17.201ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = nexus
7832023-12-06T05:05:17.204ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7842023-12-06T05:05:17.204ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7852023-12-06T05:05:17.260ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte2", _ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 151, 120]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte2" } }
7862023-12-06T05:05:17.261ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = nexus
7872023-12-06T05:05:17.263ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7882023-12-06T05:05:17.263ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7892023-12-06T05:05:17.335ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte3", _ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 138, 199]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte3" } }
7902023-12-06T05:05:17.335ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = nexus
7912023-12-06T05:05:17.337ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7922023-12-06T05:05:17.337ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7932023-12-06T05:05:17.391ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte4", _ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 173, 106]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte4" } }
7942023-12-06T05:05:17.391ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = external_dns
7952023-12-06T05:05:17.393ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7962023-12-06T05:05:17.393ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7972023-12-06T05:05:17.448ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte5", _ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 150, 34]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte5" } }
7982023-12-06T05:05:17.448ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = external_dns
7992023-12-06T05:05:17.450ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
8002023-12-06T05:05:17.450ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
8012023-12-06T05:05:17.450ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
8022023-12-06T05:05:17.450ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
8032023-12-06T05:05:17.450ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
8042023-12-06T05:05:17.450ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
8052023-12-06T05:05:17.451ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
8062023-12-06T05:05:17.451ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
8072023-12-06T05:05:17.451ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
8082023-12-06T05:05:17.451ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
8092023-12-06T05:05:17.452ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8102023-12-06T05:05:17.455ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8112023-12-06T05:05:17.455ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8122023-12-06T05:05:17.456ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8132023-12-06T05:05:17.458ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8142023-12-06T05:05:17.458ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8152023-12-06T05:05:17.458ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8162023-12-06T05:05:17.461ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8172023-12-06T05:05:17.463ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8182023-12-06T05:05:17.463ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8192023-12-06T05:05:17.618ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80 file = illumos-utils/src/zone.rs:314
8202023-12-06T05:05:17.625ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065 file = illumos-utils/src/zone.rs:314
8212023-12-06T05:05:17.632ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087 file = illumos-utils/src/zone.rs:314
8222023-12-06T05:05:17.639ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4 file = illumos-utils/src/zone.rs:314
8232023-12-06T05:05:17.647ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9 file = illumos-utils/src/zone.rs:314
8242023-12-06T05:05:17.654ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80 file = illumos-utils/src/zone.rs:353
8252023-12-06T05:05:17.662ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065 file = illumos-utils/src/zone.rs:353
8262023-12-06T05:05:17.667ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087 file = illumos-utils/src/zone.rs:353
8272023-12-06T05:05:17.673ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4 file = illumos-utils/src/zone.rs:353
8282023-12-06T05:05:17.679ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9 file = illumos-utils/src/zone.rs:353
8292023-12-06T05:05:19.531ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8302023-12-06T05:05:19.531ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
8312023-12-06T05:05:19.540ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8322023-12-06T05:05:19.540ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
8332023-12-06T05:05:19.562ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8342023-12-06T05:05:19.562ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
8352023-12-06T05:05:19.652ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="b5db29e5-d07b-473e-95d2-86b910835d56"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8362023-12-06T05:05:19.652ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
8372023-12-06T05:05:19.685ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
8382023-12-06T05:05:19.741ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="d504572e-c4c0-4f64-aa85-d13c666fb02d"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8392023-12-06T05:05:19.742ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
8402023-12-06T05:05:19.772ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="1c19b480-a707-4728-be67-e53ce6c18b88"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8412023-12-06T05:05:19.772ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
8422023-12-06T05:05:19.805ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="8a15d9f0-c96e-41de-9699-dae07090f93d"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8432023-12-06T05:05:19.805ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
8442023-12-06T05:05:19.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 27.749264ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
8452023-12-06T05:05:19.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 59.415267ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
8462023-12-06T05:05:19.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 39.66672ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
8472023-12-06T05:05:20.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 64.652158ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
8482023-12-06T05:05:20.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 46.239101ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
8492023-12-06T05:05:20.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 34.197536ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
8502023-12-06T05:05:20.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 62.956803ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
8512023-12-06T05:05:20.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 46.965384ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
8522023-12-06T05:05:20.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 52.698054ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
8532023-12-06T05:05:20.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 111.784707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
8542023-12-06T05:05:20.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 110.072398ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
8552023-12-06T05:05:20.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 149.636053ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
8562023-12-06T05:05:20.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 56.115855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
8572023-12-06T05:05:20.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 148.118017ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
8582023-12-06T05:05:20.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 124.469914ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
8592023-12-06T05:05:20.280ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
8602023-12-06T05:05:20.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 216.126238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
8612023-12-06T05:05:20.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 186.12579ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
8622023-12-06T05:05:20.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 280.946155ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
8632023-12-06T05:05:20.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 133.766062ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
8642023-12-06T05:05:20.378ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
8652023-12-06T05:05:21.385ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58249
8662023-12-06T05:05:21.386ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 182 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58249 req_id = af28d8ff-c283-42ed-959e-b5707db3268d response_code = 200 uri = /baseboard
8672023-12-06T05:05:21.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 110.059699ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
8682023-12-06T05:05:21.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 351.602943ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
8692023-12-06T05:05:21.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 415.258568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
8702023-12-06T05:05:21.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 302.294198ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
8712023-12-06T05:05:21.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 215.90855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
8722023-12-06T05:05:21.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 265.768607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
8732023-12-06T05:05:21.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 493.466687ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
8742023-12-06T05:05:21.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 237.812164ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
8752023-12-06T05:05:21.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 209.023768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
8762023-12-06T05:05:22.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 60.623672ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
8772023-12-06T05:05:22.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 872.220277ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
8782023-12-06T05:05:22.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 71.251512ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
8792023-12-06T05:05:22.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 585.846115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
8802023-12-06T05:05:22.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 852.510521ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
8812023-12-06T05:05:22.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 493.115134ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
8822023-12-06T05:05:22.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 314.663647ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
8832023-12-06T05:05:22.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 288.135343ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
8842023-12-06T05:05:22.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 326.907985ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
8852023-12-06T05:05:22.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 1.074076306s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
8862023-12-06T05:05:22.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 806.603429ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
8872023-12-06T05:05:22.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 514.573148ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
8882023-12-06T05:05:22.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 740.320687ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
8892023-12-06T05:05:22.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 828.93138ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
8902023-12-06T05:05:22.640ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_31bd71cd-4736-4a12-a387-9b74b050396f/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="5338c144-16f8-48d8-8252-af414cfb09c0"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8912023-12-06T05:05:22.640ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
8922023-12-06T05:05:22.684ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
8932023-12-06T05:05:22.706ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="b87489cc-dd39-45c8-91f9-58625b11e2f3"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8942023-12-06T05:05:22.706ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
8952023-12-06T05:05:22.728ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="6e504a86-b180-4ee1-aa51-a413404e1633"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8962023-12-06T05:05:22.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 897.895726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
8972023-12-06T05:05:22.747ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
8982023-12-06T05:05:22.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 1.110053498s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
8992023-12-06T05:05:22.781ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="0dd899a1-4a70-4a76-9f16-395eeef06f5b"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9002023-12-06T05:05:22.781ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9012023-12-06T05:05:23.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 950.549271ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
9022023-12-06T05:05:23.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 45.656487ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9032023-12-06T05:05:23.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 713.015523ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
9042023-12-06T05:05:23.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 106.115958ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9052023-12-06T05:05:23.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 65.1269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9062023-12-06T05:05:23.352ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="50b07805-cc57-4cd7-b75c-2aba9d1b988c"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9072023-12-06T05:05:23.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 985.699863ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
9082023-12-06T05:05:23.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 840.496679ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
9092023-12-06T05:05:23.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 846.972549ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
9102023-12-06T05:05:23.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 52.528785ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9112023-12-06T05:05:23.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 217.44362ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9122023-12-06T05:05:23.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 1.170787543s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
9132023-12-06T05:05:23.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 25.220335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9142023-12-06T05:05:23.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 114.142837ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9152023-12-06T05:05:23.531ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
9162023-12-06T05:05:23.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 67.473906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
9172023-12-06T05:05:23.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 647.023991ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
9182023-12-06T05:05:23.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 81.668821ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9192023-12-06T05:05:23.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 72.166985ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9202023-12-06T05:05:23.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 135.297769ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
9212023-12-06T05:05:23.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 241.440448ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9222023-12-06T05:05:23.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 646.637198ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
9232023-12-06T05:05:23.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 263.292768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9242023-12-06T05:05:23.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 103.941958ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9252023-12-06T05:05:23.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 310.602869ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9262023-12-06T05:05:23.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 213.8287ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
9272023-12-06T05:05:23.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 564.514563ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9282023-12-06T05:05:23.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 1.429516494s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
9292023-12-06T05:05:23.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 410.747219ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9302023-12-06T05:05:23.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 1.446666143s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
9312023-12-06T05:05:23.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 544.038856ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9322023-12-06T05:05:23.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 1.003468862s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
9332023-12-06T05:05:24.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 294.883543ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
9342023-12-06T05:05:24.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 797.959528ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9352023-12-06T05:05:24.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 1.231755408s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
9362023-12-06T05:05:24.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 984.198004ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
9372023-12-06T05:05:24.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 700.962336ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
9382023-12-06T05:05:24.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 1.394914016s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
9392023-12-06T05:05:24.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 1.079323606s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
9402023-12-06T05:05:24.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 803.379792ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9412023-12-06T05:05:24.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 998.377629ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
9422023-12-06T05:05:24.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 932.053946ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9432023-12-06T05:05:24.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 888.035446ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9442023-12-06T05:05:24.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 1.12035616s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
9452023-12-06T05:05:24.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 1.499954636s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9462023-12-06T05:05:25.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 1.38514488s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
9472023-12-06T05:05:25.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 837.634379ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
9482023-12-06T05:05:25.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 1.282281344s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9492023-12-06T05:05:25.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 701.333447ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
9502023-12-06T05:05:25.299ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
9512023-12-06T05:05:25.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 577.724586ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9522023-12-06T05:05:25.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 1.455982232s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
9532023-12-06T05:05:25.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 1.432925273s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
9542023-12-06T05:05:25.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 729.291796ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
9552023-12-06T05:05:25.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 1.079785322s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9562023-12-06T05:05:25.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 1.452881667s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
9572023-12-06T05:05:25.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 1.398265633s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
9582023-12-06T05:05:25.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 590.215673ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
9592023-12-06T05:05:25.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 694.387382ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
9602023-12-06T05:05:25.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 1.153793207s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
9612023-12-06T05:05:25.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 1.481161319s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9622023-12-06T05:05:25.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 759.962579ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
9632023-12-06T05:05:26.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 1.495529607s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
9642023-12-06T05:05:26.225ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
9652023-12-06T05:05:26.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 40.523171ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
9662023-12-06T05:05:26.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 941.719847ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
9672023-12-06T05:05:26.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 871.697149ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9682023-12-06T05:05:26.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 125.94134ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
9692023-12-06T05:05:26.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 1.274370531s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
9702023-12-06T05:05:26.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 875.484612ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9712023-12-06T05:05:26.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 37.859885ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
9722023-12-06T05:05:26.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 938.281404ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9732023-12-06T05:05:26.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 1.0403412s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
9742023-12-06T05:05:26.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 255.67923ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
9752023-12-06T05:05:26.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 76.832714ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
9762023-12-06T05:05:26.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 172.086293ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
9772023-12-06T05:05:26.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 948.299862ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
9782023-12-06T05:05:26.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 297.253525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
9792023-12-06T05:05:26.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 877.54883ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
9802023-12-06T05:05:26.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 1.252438497s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
9812023-12-06T05:05:26.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 234.376894ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
9822023-12-06T05:05:26.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 1.176147392s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
9832023-12-06T05:05:26.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 1.098501278s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
9842023-12-06T05:05:26.943ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
9852023-12-06T05:05:27.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 849.472212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
9862023-12-06T05:05:27.056ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
9872023-12-06T05:05:27.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 1.039537463s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
9882023-12-06T05:05:27.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 821.896715ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
9892023-12-06T05:05:27.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 52.937827ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
9902023-12-06T05:05:27.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 795.217474ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
9912023-12-06T05:05:27.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 123.785131ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
9922023-12-06T05:05:27.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 1.361870415s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
9932023-12-06T05:05:27.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 672.516486ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
9942023-12-06T05:05:27.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 44.897512ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
9952023-12-06T05:05:27.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 121.493768ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
9962023-12-06T05:05:27.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 1.128746512s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
9972023-12-06T05:05:27.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 703.447134ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
9982023-12-06T05:05:27.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 139.767393ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
9992023-12-06T05:05:27.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 326.035883ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
10002023-12-06T05:05:27.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 1.131180945s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
10012023-12-06T05:05:27.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 252.111879ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
10022023-12-06T05:05:27.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 1.383961449s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
10032023-12-06T05:05:27.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 1.071183165s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
10042023-12-06T05:05:27.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 768.63535ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
10052023-12-06T05:05:27.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 981.027135ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
10062023-12-06T05:05:27.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 938.271174ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
10072023-12-06T05:05:27.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 382.548943ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
10082023-12-06T05:05:27.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 1.070324871s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
10092023-12-06T05:05:27.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 825.430103ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
10102023-12-06T05:05:28.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 1.21827341s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
10112023-12-06T05:05:28.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 1.447208569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
10122023-12-06T05:05:28.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 1.387467357s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
10132023-12-06T05:05:28.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 762.163872ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
10142023-12-06T05:05:28.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 1.009812079s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
10152023-12-06T05:05:28.133ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10162023-12-06T05:05:28.134ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10172023-12-06T05:05:28.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 1.414648849s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
10182023-12-06T05:05:28.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 1.088831531s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
10192023-12-06T05:05:28.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 784.311738ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
10202023-12-06T05:05:28.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 37.685729ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10212023-12-06T05:05:28.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 77.389877ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10222023-12-06T05:05:28.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 681.132406ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
10232023-12-06T05:05:28.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 124.099886ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10242023-12-06T05:05:28.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 1.245737533s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
10252023-12-06T05:05:28.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 1.07229644s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
10262023-12-06T05:05:28.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 736.513266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
10272023-12-06T05:05:28.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 1.497198194s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
10282023-12-06T05:05:28.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 353.753752ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10292023-12-06T05:05:28.794ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 760.88358ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
10302023-12-06T05:05:28.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 572.920315ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
10312023-12-06T05:05:28.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 1.247535497s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
10322023-12-06T05:05:28.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 887.279611ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
10332023-12-06T05:05:29.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 1.454322123s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
10342023-12-06T05:05:29.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 1.226716015s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
10352023-12-06T05:05:29.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 856.860144ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10362023-12-06T05:05:29.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 733.418331ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
10372023-12-06T05:05:29.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 1.125768958s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
10382023-12-06T05:05:29.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 1.215919838s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
10392023-12-06T05:05:29.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 852.518773ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
10402023-12-06T05:05:29.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 772.996322ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
10412023-12-06T05:05:29.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 544.135548ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
10422023-12-06T05:05:29.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 1.401987848s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
10432023-12-06T05:05:29.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 684.221457ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
10442023-12-06T05:05:29.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 1.021622694s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
10452023-12-06T05:05:29.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 594.954942ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
10462023-12-06T05:05:29.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 895.496898ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
10472023-12-06T05:05:29.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 1.421616759s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
10482023-12-06T05:05:29.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 852.58378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
10492023-12-06T05:05:29.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 1.085281697s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
10502023-12-06T05:05:29.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 1.207158673s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
10512023-12-06T05:05:29.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 795.617726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
10522023-12-06T05:05:30.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 1.222293397s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10532023-12-06T05:05:30.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 1.470619858s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
10542023-12-06T05:05:30.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 1.002813255s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
10552023-12-06T05:05:30.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 557.045085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
10562023-12-06T05:05:30.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 775.460184ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
10572023-12-06T05:05:30.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 1.099942325s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
10582023-12-06T05:05:30.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 806.283045ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
10592023-12-06T05:05:30.322ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
10602023-12-06T05:05:30.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 1.317781306s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
10612023-12-06T05:05:30.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 701.997185ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
10622023-12-06T05:05:30.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 968.16848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
10632023-12-06T05:05:30.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 1.3381691s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
10642023-12-06T05:05:30.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 1.255302892s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
10652023-12-06T05:05:30.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 781.252668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
10662023-12-06T05:05:30.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 1.187052441s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
10672023-12-06T05:05:30.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 1.148054831s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
10682023-12-06T05:05:30.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 1.025124493s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
10692023-12-06T05:05:30.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 971.062271ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
10702023-12-06T05:05:30.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 990.332556ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
10712023-12-06T05:05:31.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 894.055747ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
10722023-12-06T05:05:31.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 1.052205474s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
10732023-12-06T05:05:31.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 615.06557ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
10742023-12-06T05:05:31.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 976.582136ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
10752023-12-06T05:05:31.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 671.734173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
10762023-12-06T05:05:31.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 1.222058609s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
10772023-12-06T05:05:31.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 776.15949ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10782023-12-06T05:05:31.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa") failed: Property not found. retry in 1.211158181s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_abbb3b13-4924-4c3e-93f6-083c065967fa
10792023-12-06T05:05:31.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 650.128638ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
10802023-12-06T05:05:31.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff") failed: Property not found. retry in 1.169564747s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_7f418c15-26fe-42f4-94b2-4bf3b2c1d2ff
10812023-12-06T05:05:31.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e") failed: Property not found. retry in 921.203275ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7e7761d-0fdf-4d42-a6b6-e5422167aa5e
10822023-12-06T05:05:31.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4") failed: Property not found. retry in 756.356886ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
10832023-12-06T05:05:31.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4") failed: Property not found. retry in 941.428307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2f0db282-de17-4fdb-8606-247b3cd829a4
10842023-12-06T05:05:31.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9") failed: Property not found. retry in 1.469152037s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
10852023-12-06T05:05:31.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389") failed: Property not found. retry in 836.168341ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e4412f1c-f03e-41c8-b0ef-e638ffd17389
10862023-12-06T05:05:31.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 524.489112ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
10872023-12-06T05:05:31.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 542.193802ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
10882023-12-06T05:05:31.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35") failed: Property not found. retry in 1.105842356s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a658ba43-3f8c-4198-9a76-2c6392304e35
10892023-12-06T05:05:31.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d") failed: Property not found. retry in 837.346236ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_41877aea-1cd2-4d10-828c-6dc0a1b00f2d
10902023-12-06T05:05:31.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25") failed: Property not found. retry in 777.085814ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8292c6ef-97da-43c9-94ae-c6763fefcb25
10912023-12-06T05:05:31.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80") failed: Property not found. retry in 1.123405753s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
10922023-12-06T05:05:31.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a") failed: Property not found. retry in 634.963709ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4eb02bef-3664-4549-bd92-a35a697fa51a
10932023-12-06T05:05:32.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a") failed: Property not found. retry in 634.847417ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_45fe60c0-65a2-4837-acb9-f15c15dda19a
10942023-12-06T05:05:32.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b") failed: Property not found. retry in 898.546094ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_646423a4-0018-4fb0-818f-0200cf3d138b
10952023-12-06T05:05:32.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d") failed: Property not found. retry in 614.586213ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
10962023-12-06T05:05:32.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065") failed: Property not found. retry in 1.230801973s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
10972023-12-06T05:05:32.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0") failed: Property not found. retry in 537.995579ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a31161d5-0c9f-4788-bd3f-3888f075c4b0
10982023-12-06T05:05:32.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087") failed: Property not found. retry in 1.006314653s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
10992023-12-06T05:05:32.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a") failed: Property not found. retry in 1.432738034s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5099d9bc-e314-42f3-a775-0218f54ecf4a
11002023-12-06T05:05:32.565ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::8 exists file = sled-agent/src/services.rs:1769
11012023-12-06T05:05:32.565ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::8, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
11022023-12-06T05:05:33.247ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::8 exists - OK file = sled-agent/src/services.rs:1777
11032023-12-06T05:05:33.248ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11042023-12-06T05:05:34.227ZINFOSledAgent (ServiceManager): Setting up external-dns service file = sled-agent/src/services.rs:1953
11052023-12-06T05:05:34.227ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_external_dns_fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4
11062023-12-06T05:05:39.293ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11072023-12-06T05:05:39.569ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::b exists file = sled-agent/src/services.rs:1769
11082023-12-06T05:05:39.569ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::b, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
11092023-12-06T05:05:40.134ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::b exists - OK file = sled-agent/src/services.rs:1777
11102023-12-06T05:05:40.134ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11112023-12-06T05:05:40.623ZINFOSledAgent (ServiceManager): Setting up Nexus service file = sled-agent/src/services.rs:1840
11122023-12-06T05:05:40.623ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_nexus_e1cbca33-5880-4c04-84b8-c054468b5065
11132023-12-06T05:05:44.297ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11142023-12-06T05:05:46.133ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::c exists file = sled-agent/src/services.rs:1769
11152023-12-06T05:05:46.133ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::c, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
11162023-12-06T05:05:46.690ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::c exists - OK file = sled-agent/src/services.rs:1777
11172023-12-06T05:05:46.690ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11182023-12-06T05:05:47.138ZINFOSledAgent (ServiceManager): Setting up Nexus service file = sled-agent/src/services.rs:1840
11192023-12-06T05:05:47.138ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_nexus_983419ea-e2f5-4b1a-ac1a-57f106744087
11202023-12-06T05:05:49.300ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11212023-12-06T05:05:51.462ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:65469
11222023-12-06T05:05:51.463ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 770 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:65469 req_id = a736e024-367b-456e-ad98-6aba14082f0b response_code = 200 uri = /baseboard
11232023-12-06T05:05:52.863ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::d exists file = sled-agent/src/services.rs:1769
11242023-12-06T05:05:52.864ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::d, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_oximeter_46c23f11-8078-4578-83a9-a198e53b593d
11252023-12-06T05:05:53.429ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::d exists - OK file = sled-agent/src/services.rs:1777
11262023-12-06T05:05:53.429ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11272023-12-06T05:05:54.302ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11282023-12-06T05:05:54.424ZINFOSledAgent (ServiceManager): Setting up oximeter service file = sled-agent/src/services.rs:2064
11292023-12-06T05:05:54.617ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::9 exists file = sled-agent/src/services.rs:1769
11302023-12-06T05:05:54.617ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::9, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
11312023-12-06T05:05:55.179ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::9 exists - OK file = sled-agent/src/services.rs:1777
11322023-12-06T05:05:55.179ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11332023-12-06T05:05:56.184ZINFOSledAgent (ServiceManager): Setting up external-dns service file = sled-agent/src/services.rs:1953
11342023-12-06T05:05:56.184ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_external_dns_72cbd99a-a71b-4108-ac9e-b67ccdec1db9
11352023-12-06T05:05:59.304ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11362023-12-06T05:06:02.669ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::a exists file = sled-agent/src/services.rs:1769
11372023-12-06T05:06:02.669ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::a, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
11382023-12-06T05:06:03.234ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::a exists - OK file = sled-agent/src/services.rs:1777
11392023-12-06T05:06:03.235ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11402023-12-06T05:06:03.518ZINFOSledAgent (ServiceManager): Setting up Nexus service file = sled-agent/src/services.rs:1840
11412023-12-06T05:06:03.518ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_nexus_8dde7bf4-95ce-4d31-8b97-4393ef169a80
11422023-12-06T05:06:04.305ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11432023-12-06T05:06:08.584ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
11442023-12-06T05:06:08.584ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
11452023-12-06T05:06:08.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 53306472 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64164 req_id = add71dbc-0837-4296-98ca-7a1668e0ebe8 response_code = 204 uri = /omicron-zones
11462023-12-06T05:06:08.589ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1062
11472023-12-06T05:06:08.589ZINFOSledAgent (RSS): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:198
11482023-12-06T05:06:08.589ZINFOSledAgent (RSS): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:198
11492023-12-06T05:06:08.592ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:549
11502023-12-06T05:06:08.592ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::a]:12221 file = sled-agent/src/rack_setup/service.rs:551
11512023-12-06T05:06:08.624ZINFOSledAgent (RSS): rack_network_config: Some( RackNetworkConfigV1 { bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { addresses: [ V4( Ipv4Network { addr: 10.151.2.174, prefix: 32, }, ), ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.2.100, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Network { addr: fd00:1122:3344:1::, prefix: 56, }, }, ) file = sled-agent/src/rack_setup/service.rs:658
11522023-12-06T05:06:08.625ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/9fd18c44-b5ff-4007-bd14-a497b2fbd8ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11532023-12-06T05:06:08.754ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/9fd18c44-b5ff-4007-bd14-a497b2fbd8ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11542023-12-06T05:06:08.879ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/9fd18c44-b5ff-4007-bd14-a497b2fbd8ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11552023-12-06T05:06:09.036ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/9fd18c44-b5ff-4007-bd14-a497b2fbd8ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11562023-12-06T05:06:09.308ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11572023-12-06T05:06:10.344ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a2c1290d-cc78-4474-9931-302132a99269", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a2c1290d-cc78-4474-9931-302132a99269" } file = sled-agent/src/rack_setup/service.rs:679
11582023-12-06T05:06:10.664ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6db95546-d3de-45e0-94c8-d2582e513018", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6db95546-d3de-45e0-94c8-d2582e513018" } file = sled-agent/src/rack_setup/service.rs:679
11592023-12-06T05:06:11.005ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ff48f2d4-584d-46a4-9888-85b37cc576ee", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ff48f2d4-584d-46a4-9888-85b37cc576ee" } file = sled-agent/src/rack_setup/service.rs:679
11602023-12-06T05:06:11.227ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "032ddb0c-821a-4c3f-964b-aa44d8ca68f4", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "032ddb0c-821a-4c3f-964b-aa44d8ca68f4" } file = sled-agent/src/rack_setup/service.rs:679
11612023-12-06T05:06:11.832ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3c3466ba-9f8a-4611-aef0-d7fa9690e121", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3c3466ba-9f8a-4611-aef0-d7fa9690e121" } file = sled-agent/src/rack_setup/service.rs:679
11622023-12-06T05:06:12.403ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5e9482da-1568-4b55-ac6e-110bba7472cd", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5e9482da-1568-4b55-ac6e-110bba7472cd" } file = sled-agent/src/rack_setup/service.rs:679
11632023-12-06T05:06:12.950ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "09a3c433-84f0-4ee5-a114-90fd1de3d420", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "09a3c433-84f0-4ee5-a114-90fd1de3d420" } file = sled-agent/src/rack_setup/service.rs:679
11642023-12-06T05:06:13.643ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2ba0a473-08a8-4d5e-9fb6-be5034a7e0f2", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2ba0a473-08a8-4d5e-9fb6-be5034a7e0f2" } file = sled-agent/src/rack_setup/service.rs:679
11652023-12-06T05:06:14.117ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "731d26bb-a931-4e71-b520-4f5535453b60", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "731d26bb-a931-4e71-b520-4f5535453b60" } file = sled-agent/src/rack_setup/service.rs:679
11662023-12-06T05:06:14.310ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11672023-12-06T05:06:15.190ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8983b3d3-e168-43ab-8582-f44ecd07487b", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8983b3d3-e168-43ab-8582-f44ecd07487b" } file = sled-agent/src/rack_setup/service.rs:679
11682023-12-06T05:06:16.124ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5c7c00d4-275a-4703-95e6-4c942524d867", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5c7c00d4-275a-4703-95e6-4c942524d867" } file = sled-agent/src/rack_setup/service.rs:679
11692023-12-06T05:06:17.035ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "37fcec40-e340-4455-9b03-47f3f885373d", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "37fcec40-e340-4455-9b03-47f3f885373d" } file = sled-agent/src/rack_setup/service.rs:679
11702023-12-06T05:06:19.312ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11712023-12-06T05:06:19.370ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "270d24c5-ffff-4777-80ae-0883cd260f1b", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "270d24c5-ffff-4777-80ae-0883cd260f1b" } file = sled-agent/src/rack_setup/service.rs:679
11722023-12-06T05:06:21.498ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45055
11732023-12-06T05:06:21.498ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 40 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45055 req_id = 8c25f85e-9d7d-486a-b6b4-e093fe08e0e5 response_code = 200 uri = /baseboard
11742023-12-06T05:06:21.781ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e6b6b9a7-3d48-4c2e-a03a-c1bf98f16879", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:21 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e6b6b9a7-3d48-4c2e-a03a-c1bf98f16879" } file = sled-agent/src/rack_setup/service.rs:679
11752023-12-06T05:06:24.314ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11762023-12-06T05:06:24.362ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9880322d-aeda-4c19-8a38-281547e3b699", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9880322d-aeda-4c19-8a38-281547e3b699" } file = sled-agent/src/rack_setup/service.rs:679
11772023-12-06T05:06:26.074ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5276d1a6-23d5-4a0b-926b-a079d6226cde", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:26 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5276d1a6-23d5-4a0b-926b-a079d6226cde" } file = sled-agent/src/rack_setup/service.rs:679
11782023-12-06T05:06:28.436ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4414763f-aa80-444e-84e8-f5d908f254e3", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:28 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4414763f-aa80-444e-84e8-f5d908f254e3" } file = sled-agent/src/rack_setup/service.rs:679
11792023-12-06T05:06:29.316ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11802023-12-06T05:06:30.797ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9bf546fe-63f0-4999-a93a-dc3796f1447d", "content-length": "124", "date": "Wed, 06 Dec 2023 05:06:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9bf546fe-63f0-4999-a93a-dc3796f1447d" } file = sled-agent/src/rack_setup/service.rs:679
11812023-12-06T05:06:33.858ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"a1bf0175-91e3-458d-b255-58082194bc92\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Wed, 06 Dec 2023 05:06:33 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"a1bf0175-91e3-458d-b255-58082194bc92\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 157.419108374s
11822023-12-06T05:06:33.888ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"b0a168a9-d734-4986-9191-cf3e97bef85d\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Wed, 06 Dec 2023 05:06:33 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"b0a168a9-d734-4986-9191-cf3e97bef85d\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 157.450259875s
11832023-12-06T05:06:34.015ZINFOSledAgent (HardwareMonitor): contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "000c3dd0fcba4d4493e687d9695a9876" } file = sled-agent/src/sled_agent.rs:629 id = 81c25e4c-3681-431d-a398-2d4416a62095
11842023-12-06T05:06:34.036ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"f3089e2a-21c4-40ec-afe9-ea83f949da9f\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Wed, 06 Dec 2023 05:06:34 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"f3089e2a-21c4-40ec-afe9-ea83f949da9f\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 157.597799715s
11852023-12-06T05:06:34.071ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"43936ad2-ab53-48d5-8be7-0eb9e1f83734\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Wed, 06 Dec 2023 05:06:33 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"43936ad2-ab53-48d5-8be7-0eb9e1f83734\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 157.63217973s
11862023-12-06T05:06:34.074ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11872023-12-06T05:06:34.081ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11882023-12-06T05:06:34.081ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11892023-12-06T05:06:34.084ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"e8a5969d-3f7e-4014-a5ee-7bcf6968e787\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Wed, 06 Dec 2023 05:06:33 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"e8a5969d-3f7e-4014-a5ee-7bcf6968e787\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 157.645235909s
11902023-12-06T05:06:34.087ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11912023-12-06T05:06:34.096ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"960017fd-7cea-4b18-8a62-25c1f7464bfa\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Wed, 06 Dec 2023 05:06:34 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"960017fd-7cea-4b18-8a62-25c1f7464bfa\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 157.657097357s
11922023-12-06T05:06:34.098ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: M2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11932023-12-06T05:06:34.104ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: M2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11942023-12-06T05:06:34.114ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11952023-12-06T05:06:34.114ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11962023-12-06T05:06:34.126ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11972023-12-06T05:06:34.126ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11982023-12-06T05:06:34.137ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: 81c25e4c-3681-431d-a398-2d4416a62095, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11992023-12-06T05:06:34.318ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12002023-12-06T05:06:34.880ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "e777ee84-75dc-4715-872e-ce516e251587", "content-length": "174", "date": "Wed, 06 Dec 2023 05:06:34 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: zpool with id \"14b4dc87-ab46-49fb-a4b4-d361ae214c03\"", request_id: "e777ee84-75dc-4715-872e-ce516e251587" } file = sled-agent/src/rack_setup/service.rs:679
12012023-12-06T05:06:36.536ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12022023-12-06T05:06:36.536ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12032023-12-06T05:06:36.536ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12042023-12-06T05:06:36.536ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12052023-12-06T05:06:39.319ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12062023-12-06T05:06:39.349ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "432202b2-2261-4ae8-82c3-bebdaad5a1be", "content-length": "174", "date": "Wed, 06 Dec 2023 05:06:39 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: zpool with id \"cd70d7f6-2354-4bf2-8012-55bf9eaf7930\"", request_id: "432202b2-2261-4ae8-82c3-bebdaad5a1be" } file = sled-agent/src/rack_setup/service.rs:679
12072023-12-06T05:06:39.365ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12082023-12-06T05:06:39.365ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12092023-12-06T05:06:40.807ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12102023-12-06T05:06:40.807ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12112023-12-06T05:06:40.807ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12122023-12-06T05:06:40.807ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12132023-12-06T05:06:40.807ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
12142023-12-06T05:06:44.321ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12152023-12-06T05:06:46.436ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38769
12162023-12-06T05:06:46.437ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:399 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13c5ab6d-2c8e-458a-bd69-abf74fb65618, kind: Service { id: 72cbd99a-a71b-4108-ac9e-b67ccdec1db9 }, name: Name("external-dns-72cbd99a-a71b-4108-ac9e-b67ccdec1db9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 34])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5f7538d9-7dd9-4257-a4dd-751c9f1af140, kind: Service { id: fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4 }, name: Name("external-dns-fd0792cb-95bd-40a2-a8a7-01b57cd0bfb4"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 173, 106])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0d17d4ea-9aa2-43b9-8a3c-7f7650f77c7d, kind: Service { id: e1cbca33-5880-4c04-84b8-c054468b5065 }, name: Name("nexus-e1cbca33-5880-4c04-84b8-c054468b5065"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 120])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b364d85f-e703-447f-98b5-e0cc6ff589d3, kind: Service { id: 8dde7bf4-95ce-4d31-8b97-4393ef169a80 }, name: Name("nexus-8dde7bf4-95ce-4d31-8b97-4393ef169a80"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 172, 159])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: cb9210f0-f677-49dd-8af4-6c86b07d954d, kind: Service { id: 983419ea-e2f5-4b1a-ac1a-57f106744087 }, name: Name("nexus-983419ea-e2f5-4b1a-ac1a-57f106744087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 199])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
12172023-12-06T05:06:46.437ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
12182023-12-06T05:06:46.437ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
12192023-12-06T05:06:46.437ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
12202023-12-06T05:06:46.437ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
12212023-12-06T05:06:46.437ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
12222023-12-06T05:06:46.437ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte0" rules = []
12232023-12-06T05:06:46.437ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 391 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38769 req_id = 0fdac447-b36f-426c-983b-b2d9fec7d78a response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
12242023-12-06T05:06:46.726ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42810
12252023-12-06T05:06:46.726ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 670 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42810 req_id = 63617427-04c1-44bd-bdc1-645f09429c25 response_code = 200 uri = /network-bootstore-config
12262023-12-06T05:06:46.732ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:689
12272023-12-06T05:06:49.323ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12282023-12-06T05:06:51.534ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46626
12292023-12-06T05:06:51.534ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 32 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46626 req_id = 90dede2b-fba0-4bc0-a7c1-f91d29283658 response_code = 200 uri = /baseboard
12302023-12-06T05:06:54.325ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12312023-12-06T05:06:57.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43458
12322023-12-06T05:06:57.961ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:227 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: "localshark" }, nics: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, external_ips: [10.151.2.176], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(9648298))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-c78419fc78d", read_only: false, slot: Slot(1), volume_construction_request: Volume { block_size: 512, id: fa3d3741-ec6a-49af-b96a-c61707269970, read_only_parent: Some(Url { block_size: 512, id: 45bac71c-6eed-44e5-b345-c495199751d2, url: "http://[fd00:1122:3344:101::1]:54321/debian-11-genericcloud-amd64.raw" }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: fa3d3741-ec6a-49af-b96a-c61707269970, key: Some("gEY4Uqa2s4Eo6ZBimwQLI1xRy9xAvLzd3XyLE1N7mK4="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19000", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::12]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAADA3CiGV4ZXAADcKIZXAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAwNwohleGVwAA3CiGVwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzZjQ4ZGJkZC1lZDM5LTQ0ZmUtYjM4NC05MjBhYjI1ODBiNjAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRGVXQmxKQ2lzS3VQWEl4VmdKOVdHcXFwczNaY3dSMmZrQ1N6K3h0bHBSWSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 instance_runtime = InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z } propolis_addr = [fd00:1122:3344:101::22]:12400 propolis_id = 1189392e-ce12-4e85-8fb0-f4e42486c845 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2023-12-06T05:06:57.554753Z }
12332023-12-06T05:06:57.961ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:261 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12342023-12-06T05:06:57.961ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:599 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 propolis_id = 1189392e-ce12-4e85-8fb0-f4e42486c845 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: "localshark" }, nics: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, external_ips: [10.151.2.176], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(9648298))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6ae87c26-9c1e-4a37-88bf-d8553d308bef, kind: Instance { id: 3f48dbdd-ed39-44fe-b384-920ab2580b60 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 242, 198, 57])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(9648298), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-c78419fc78d", read_only: false, slot: Slot(1), volume_construction_request: Volume { block_size: 512, id: fa3d3741-ec6a-49af-b96a-c61707269970, read_only_parent: Some(Url { block_size: 512, id: 45bac71c-6eed-44e5-b345-c495199751d2, url: "http://[fd00:1122:3344:101::1]:54321/debian-11-genericcloud-amd64.raw" }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: fa3d3741-ec6a-49af-b96a-c61707269970, key: Some("gEY4Uqa2s4Eo6ZBimwQLI1xRy9xAvLzd3XyLE1N7mK4="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19000", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::12]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAADA3CiGV4ZXAADcKIZXAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAwNwohleGVwAA3CiGVwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzZjQ4ZGJkZC1lZDM5LTQ0ZmUtYjM4NC05MjBhYjI1ODBiNjAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRGVXQmxKQ2lzS3VQWEl4VmdKOVdHcXFwczNaY3dSMmZrQ1N6K3h0bHBSWSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2023-12-06T05:06:57.554753Z }, propolis_addr: [fd00:1122:3344:101::22]:12400 }
12352023-12-06T05:06:57.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 1279 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43458 req_id = 1f607f51-729b-463f-b42b-4ab2176d76fb response_code = 200 uri = /instances/3f48dbdd-ed39-44fe-b384-920ab2580b60
12362023-12-06T05:06:58.054ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52555
12372023-12-06T05:06:58.075ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte6", _ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 242, 198, 57]), slot: 0, vni: Vni { inner: 9648298 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
12382023-12-06T05:06:58.076ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
12392023-12-06T05:06:58.111ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845 file = illumos-utils/src/zone.rs:314 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12402023-12-06T05:06:58.133ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845 file = illumos-utils/src/zone.rs:353 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12412023-12-06T05:06:59.327ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12422023-12-06T05:07:00.265ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::22"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::b]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12432023-12-06T05:07:00.266ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12442023-12-06T05:07:00.452ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 69.565804ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12452023-12-06T05:07:00.536ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 66.767634ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12462023-12-06T05:07:00.616ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 192.707586ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12472023-12-06T05:07:00.821ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 325.831507ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12482023-12-06T05:07:01.158ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 487.457319ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12492023-12-06T05:07:01.657ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 1.453971725s file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12502023-12-06T05:07:03.121ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 1.003369603s file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12512023-12-06T05:07:03.998ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::d]:40179
12522023-12-06T05:07:04.000ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 1748 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:40179 req_id = a14d8002-1028-4fbd-8407-f447590112ad response_code = 200 uri = /metrics/collect/81c25e4c-3681-431d-a398-2d4416a62095
12532023-12-06T05:07:04.134ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 604.732007ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12542023-12-06T05:07:04.328ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12552023-12-06T05:07:04.752ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 598.046871ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 zone = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
12562023-12-06T05:07:05.379ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845 file = sled-agent/src/instance.rs:986 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12572023-12-06T05:07:05.389ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 41.244478ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12582023-12-06T05:07:05.439ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845") failed: Property not found. retry in 127.134875ms file = illumos-utils/src/svc.rs:36 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12592023-12-06T05:07:05.578ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:995 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12602023-12-06T05:07:05.612ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1009 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12612023-12-06T05:07:05.613ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAADA3CiGV4ZXAADcKIZXAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAwNwohleGVwAA3CiGVwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzZjQ4ZGJkZC1lZDM5LTQ0ZmUtYjM4NC05MjBhYjI1ODBiNjAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRGVXQmxKQ2lzS3VQWEl4VmdKOVdHcXFwczNaY3dSMmZrQ1N6K3h0bHBSWSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-c78419fc78d", read_only: false, slot: Slot(1), volume_construction_request: Volume { block_size: 512, id: fa3d3741-ec6a-49af-b96a-c61707269970, read_only_parent: Some(Url { block_size: 512, id: 45bac71c-6eed-44e5-b345-c495199751d2, url: "http://[fd00:1122:3344:101::1]:54321/debian-11-genericcloud-amd64.raw" }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: fa3d3741-ec6a-49af-b96a-c61707269970, key: Some("gEY4Uqa2s4Eo6ZBimwQLI1xRy9xAvLzd3XyLE1N7mK4="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19000", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::12]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 3f48dbdd-ed39-44fe-b384-920ab2580b60, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:450 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12622023-12-06T05:07:05.829ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:452 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12632023-12-06T05:07:05.829ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7775190 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52555 req_id = 4dbc3fcb-9739-4d4d-ae41-517ccbdd10d5 response_code = 200 uri = /instances/3f48dbdd-ed39-44fe-b384-920ab2580b60/state
12642023-12-06T05:07:05.829ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2023-12-06T05:07:05.829880361Z } file = sled-agent/src/instance.rs:334 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12652023-12-06T05:07:05.829ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 new_instance_state = InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2023-12-06T05:07:05.829880361Z } propolis_id = 1189392e-ce12-4e85-8fb0-f4e42486c845
12662023-12-06T05:07:05.829ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z }, propolis_id: 1189392e-ce12-4e85-8fb0-f4e42486c845, vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2023-12-06T05:07:05.829880361Z } }
12672023-12-06T05:07:05.877ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2023-12-06T05:07:05.877257053Z } file = sled-agent/src/instance.rs:334 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
12682023-12-06T05:07:05.877ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 new_instance_state = InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2023-12-06T05:07:05.877257053Z } propolis_id = 1189392e-ce12-4e85-8fb0-f4e42486c845
12692023-12-06T05:07:05.877ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z }, propolis_id: 1189392e-ce12-4e85-8fb0-f4e42486c845, vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2023-12-06T05:07:05.877257053Z } }
12702023-12-06T05:07:09.330ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12712023-12-06T05:07:14.331ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12722023-12-06T05:07:19.333ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12732023-12-06T05:07:21.569ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58462
12742023-12-06T05:07:21.570ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 777 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58462 req_id = 44ecc6dd-d028-4f6f-83f1-69f7bb9b7c0c response_code = 200 uri = /baseboard
12752023-12-06T05:07:24.334ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12762023-12-06T05:07:29.337ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12772023-12-06T05:07:33.999ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 915 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:40179 req_id = 294bd12a-e403-4bb0-9367-ef6f5f870b8f response_code = 200 uri = /metrics/collect/81c25e4c-3681-431d-a398-2d4416a62095
12782023-12-06T05:07:34.338ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12792023-12-06T05:07:39.339ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12802023-12-06T05:07:44.340ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12812023-12-06T05:07:49.342ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12822023-12-06T05:07:51.604ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:49394
12832023-12-06T05:07:51.604ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 50 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:49394 req_id = e94ee515-7134-4e57-b0ed-db70b78b0a76 response_code = 200 uri = /baseboard
12842023-12-06T05:07:54.344ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12852023-12-06T05:07:59.344ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12862023-12-06T05:08:03.999ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 380 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:40179 req_id = 29dd4238-5141-463f-b514-6c16a4e1275a response_code = 200 uri = /metrics/collect/81c25e4c-3681-431d-a398-2d4416a62095
12872023-12-06T05:08:04.346ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12882023-12-06T05:08:09.348ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12892023-12-06T05:08:14.350ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12902023-12-06T05:08:19.351ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12912023-12-06T05:08:21.638ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:54793
12922023-12-06T05:08:21.638ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 32 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:54793 req_id = 4014a76c-6d56-42ce-9d2f-a64e43323d17 response_code = 200 uri = /baseboard
12932023-12-06T05:08:24.353ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12942023-12-06T05:08:29.355ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12952023-12-06T05:08:33.998ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:40179 req_id = 9e0bdcd5-c958-4ac8-b123-17b8682a20be response_code = 200 uri = /metrics/collect/81c25e4c-3681-431d-a398-2d4416a62095
12962023-12-06T05:08:34.357ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12972023-12-06T05:08:39.358ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12982023-12-06T05:08:44.359ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12992023-12-06T05:08:49.360ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13002023-12-06T05:08:51.673ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46327
13012023-12-06T05:08:51.673ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 43 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46327 req_id = 16ea3c24-13b9-4638-8d53-70e198ff44cb response_code = 200 uri = /baseboard
13022023-12-06T05:08:54.361ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13032023-12-06T05:08:59.362ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13042023-12-06T05:09:03.998ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 267 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:40179 req_id = 30bdff2b-9b82-4d7e-901d-90ee0b9fd044 response_code = 200 uri = /metrics/collect/81c25e4c-3681-431d-a398-2d4416a62095
13052023-12-06T05:09:04.364ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13062023-12-06T05:09:09.365ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13072023-12-06T05:09:14.367ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13082023-12-06T05:09:19.368ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13092023-12-06T05:09:21.706ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:38860
13102023-12-06T05:09:21.706ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 36 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:38860 req_id = 161d6cd9-9e15-426a-99ff-f1440d0ff170 response_code = 200 uri = /baseboard
13112023-12-06T05:09:24.370ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13122023-12-06T05:09:29.371ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13132023-12-06T05:09:33.999ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:40179 req_id = 796eec55-084c-4381-875d-4f24c4c270e6 response_code = 200 uri = /metrics/collect/81c25e4c-3681-431d-a398-2d4416a62095
13142023-12-06T05:09:34.373ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13152023-12-06T05:09:39.375ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13162023-12-06T05:09:44.376ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13172023-12-06T05:09:47.156ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59283
13182023-12-06T05:09:47.157ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 453 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59283 req_id = 4b4a6a9d-c1d3-493d-974b-3f4db48a7ff1 response_code = 200 uri = /instances/3f48dbdd-ed39-44fe-b384-920ab2580b60/state
13192023-12-06T05:09:47.157ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2023-12-06T05:09:47.157529179Z } file = sled-agent/src/instance.rs:334 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
13202023-12-06T05:09:47.157ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 new_instance_state = InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2023-12-06T05:09:47.157529179Z } propolis_id = 1189392e-ce12-4e85-8fb0-f4e42486c845
13212023-12-06T05:09:47.157ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z }, propolis_id: 1189392e-ce12-4e85-8fb0-f4e42486c845, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2023-12-06T05:09:47.157529179Z } }
13222023-12-06T05:09:47.179ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopped), migration_status: NoMigration, time: 2023-12-06T05:09:47.179016369Z } file = sled-agent/src/instance.rs:334 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
13232023-12-06T05:09:47.179ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 new_instance_state = InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2023-12-06T05:09:47.179016369Z } propolis_id = 1189392e-ce12-4e85-8fb0-f4e42486c845
13242023-12-06T05:09:47.179ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(1189392e-ce12-4e85-8fb0-f4e42486c845), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-06T05:06:57.604966210Z }, propolis_id: 1189392e-ce12-4e85-8fb0-f4e42486c845, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2023-12-06T05:09:47.179016369Z } }
13252023-12-06T05:09:47.353ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2023-12-06T05:09:47.353488779Z } file = sled-agent/src/instance.rs:334 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
13262023-12-06T05:09:47.353ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2023-12-06T05:09:47.353488779Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2023-12-06T05:09:47.353488779Z } propolis_id = 1189392e-ce12-4e85-8fb0-f4e42486c845
13272023-12-06T05:09:47.353ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:371 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
13282023-12-06T05:09:47.353ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
13292023-12-06T05:09:47.354ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
13302023-12-06T05:09:47.354ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/debug/bundle/zone", "/pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug", "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug", "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/debug", "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/debug", "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/debug", "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/debug", "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/debug", "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug", "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:436 zone_name = oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845
13312023-12-06T05:09:48.006ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1229 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845", bundle_id: 1069d72e-4b7d-48f1-bcde-cf8632a2a85c }, time_created: 2023-12-06T05:09:47.354358754Z, version: 0, cause: TerminatedInstance }
13322023-12-06T05:09:48.006ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_1189392e-ce12-4e85-8fb0-f4e42486c845 file = sled-agent/src/instance.rs:544 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
13332023-12-06T05:09:49.376ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::94fa:6dff:fe2d:d7df": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13342023-12-06T05:09:49.580ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:268 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
13352023-12-06T05:09:49.615ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2023-12-06T05:09:47.353488779Z }, propolis_id: 1189392e-ce12-4e85-8fb0-f4e42486c845, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2023-12-06T05:09:47.353488779Z } }
13362023-12-06T05:09:49.749ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57079
13372023-12-06T05:09:49.749ZWARNSledAgent (PortManager): unset_virtual_nic_host unimplmented file = illumos-utils/src/opte/port_manager.rs:494
13382023-12-06T05:09:49.749ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 187 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:57079 req_id = 1310fa41-e9ff-4746-bb33-19cd08ddfeda response_code = 204 uri = /v2p/6ae87c26-9c1e-4a37-88bf-d8553d308bef
13392023-12-06T05:09:49.867ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:487 instance_id = 3f48dbdd-ed39-44fe-b384-920ab2580b60
13402023-12-06T05:09:51.741ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40514
13412023-12-06T05:09:51.741ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 44 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40514 req_id = 7354ed22-b097-4a5f-8a1c-c26ccf304e0e response_code = 200 uri = /baseboard