Max level shown:
1 [ Dec 5 05:22:42 Enabled. ]
2 [ Dec 5 05:22:42 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Dec 5 05:22:42 Method "start" exited with status 0. ]
4 [ Dec 5 05:22:42 Rereading configuration. ]
5 [ Dec 5 05:22:42 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72023-12-05T05:22:42.340ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:240
82023-12-05T05:22:42.394ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:247
92023-12-05T05:22:42.394ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
102023-12-05T05:22:42.502ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
112023-12-05T05:22:42.511ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
122023-12-05T05:22:42.520ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:257 path = /var/run/oxide/
132023-12-05T05:22:43.123ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:9fb2::, len: 64 }
142023-12-05T05:22:43.262ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:117
152023-12-05T05:22:43.262ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:129
162023-12-05T05:22:43.262ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
172023-12-05T05:22:43.262ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 sled_mode = Scrimlet { asic: SoftNpuZone }
182023-12-05T05:22:43.262ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:606
192023-12-05T05:22:43.262ZINFOSledAgent (DumpSetup-archival): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:171
202023-12-05T05:22:43.262ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
212023-12-05T05:22:43.265ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "e0b9a1778185441db38d4af540b43da0", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:532
222023-12-05T05:22:43.265ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:655
232023-12-05T05:22:43.265ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
242023-12-05T05:22:43.265ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:713
252023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b file = sled-agent/src/long_running_tasks.rs:232
262023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b file = sled-agent/src/long_running_tasks.rs:232
272023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b file = sled-agent/src/long_running_tasks.rs:232
282023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
292023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
302023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
312023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 file = sled-agent/src/long_running_tasks.rs:232
322023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 file = sled-agent/src/long_running_tasks.rs:232
332023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 file = sled-agent/src/long_running_tasks.rs:232
342023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_31bd71cd-4736-4a12-a387-9b74b050396f file = sled-agent/src/long_running_tasks.rs:232
352023-12-05T05:22:43.265ZINFOSledAgent: Upserting synthetic zpool to Storage Manager: oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 file = sled-agent/src/long_running_tasks.rs:232
362023-12-05T05:22:43.265ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:88
372023-12-05T05:22:43.265ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:241 underlay_network_info = None
382023-12-05T05:22:43.265ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } })) file = sled-storage/src/manager.rs:346
392023-12-05T05:22:43.449ZINFOSledAgent (StorageManager): Channel capacity decreased current = 245 file = sled-storage/src/manager.rs:326 previous = 256
402023-12-05T05:22:43.451ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } file = sled-agent/src/long_running_tasks.rs:90
412023-12-05T05:22:43.451ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } })) file = sled-storage/src/manager.rs:346
422023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External } })) file = sled-storage/src/manager.rs:346
432023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
442023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
452023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
462023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } })) file = sled-storage/src/manager.rs:346
472023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" }, zpool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } })) file = sled-storage/src/manager.rs:346
482023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" }, zpool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External } })) file = sled-storage/src/manager.rs:346
492023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" }, zpool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } })) file = sled-storage/src/manager.rs:346
502023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received AddDisk(Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" }, zpool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } })) file = sled-storage/src/manager.rs:346
512023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received DisksChanged({}) file = sled-storage/src/manager.rs:346
522023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Channel capacity increased current = 255 file = sled-storage/src/manager.rs:336 previous = 245
532023-12-05T05:22:43.621ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
542023-12-05T05:22:43.621ZINFOSledAgent (StorageMonitor): Received storage manager update file = sled-agent/src/storage_monitor.rs:111 resources = StorageResources { disks: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 497664, free: 20937467904, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 497664, free: 20937467904, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } })} }
552023-12-05T05:22:43.621ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
562023-12-05T05:22:43.621ZWARNSledAgent (bootstore): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:187 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
572023-12-05T05:22:43.621ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
582023-12-05T05:22:43.621ZWARNSledAgent (bootstore): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:187 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
592023-12-05T05:22:43.621ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-fsm-state.json peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
602023-12-05T05:22:43.621ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
612023-12-05T05:22:43.621ZWARNSledAgent (bootstore): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json file = common/src/ledger.rs:187 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
622023-12-05T05:22:43.621ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
632023-12-05T05:22:43.621ZWARNSledAgent (bootstore): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json file = common/src/ledger.rs:187 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
642023-12-05T05:22:43.621ZERROSledAgent (bootstore): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/bootstore-network-config.json peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
652023-12-05T05:22:43.621ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
662023-12-05T05:22:43.621ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
672023-12-05T05:22:43.621ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
682023-12-05T05:22:43.636ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:575
692023-12-05T05:22:43.636ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3042
702023-12-05T05:22:43.636ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:288 next_cleanup = Instant { tv_sec: 1273, tv_nsec: 956227911 } time_to_next_cleanup = 599.999920068s
712023-12-05T05:22:43.636ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
722023-12-05T05:22:43.636ZWARNSledAgent (BootstrapAgentStartup): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json file = common/src/ledger.rs:187
732023-12-05T05:22:43.636ZERROSledAgent (BootstrapAgentStartup): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json
742023-12-05T05:22:43.639ZWARNSledAgent (BootstrapAgentStartup): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json file = common/src/ledger.rs:187
752023-12-05T05:22:43.639ZERROSledAgent (BootstrapAgentStartup): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/sled-agent-request.json
762023-12-05T05:22:43.639ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:195 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
772023-12-05T05:22:43.639ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
782023-12-05T05:22:43.640ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { rack_subnet: fd00:1122:3344:100::, trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: Some( RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:1::, prefix: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.1.100, }, ], addresses: [ V4( Ipv4Network { addr: 10.151.1.174, prefix: 32, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], }, ), } file = sled-agent/src/rack_setup/service.rs:803
792023-12-05T05:22:43.640ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
802023-12-05T05:22:43.640ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
812023-12-05T05:22:43.640ZWARNSledAgent (RSS): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker file = common/src/ledger.rs:187
822023-12-05T05:22:43.640ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker
832023-12-05T05:22:43.640ZWARNSledAgent (RSS): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker file = common/src/ledger.rs:187
842023-12-05T05:22:43.640ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-plan-completed.marker
852023-12-05T05:22:43.640ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:863
862023-12-05T05:22:43.640ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
872023-12-05T05:22:43.640ZWARNSledAgent (RSS): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json file = common/src/ledger.rs:187
882023-12-05T05:22:43.640ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json
892023-12-05T05:22:43.640ZWARNSledAgent (RSS): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json file = common/src/ledger.rs:187
902023-12-05T05:22:43.640ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-sled-plan.json
912023-12-05T05:22:43.640ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:904
922023-12-05T05:22:43.640ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
932023-12-05T05:22:43.640ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
942023-12-05T05:22:43.640ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
952023-12-05T05:22:43.640ZINFOSledAgent (RSS): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:198
962023-12-05T05:22:43.641ZINFOSledAgent (RSS): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:198
972023-12-05T05:22:43.641ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
982023-12-05T05:22:43.641ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:937
992023-12-05T05:22:43.641ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:556 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
1002023-12-05T05:22:43.641ZINFOSledAgent (bootstore): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:198 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
1012023-12-05T05:22:43.641ZINFOSledAgent (bootstore): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:198 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
1022023-12-05T05:22:43.641ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:640 peer_id = pc-e0b9a1778185441db38d4af540b43da0-i86pc
1032023-12-05T05:22:43.641ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, rack_id: 7f701f68-f023-4c8b-84b4-4cfebb9ea428, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1042023-12-05T05:22:43.641ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:56047
1052023-12-05T05:22:43.641ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, rack_id: 7f701f68-f023-4c8b-84b4-4cfebb9ea428, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } file = sled-agent/src/bootstrap/server.rs:364
1062023-12-05T05:22:43.641ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:379
1072023-12-05T05:22:43.641ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:47
1082023-12-05T05:22:43.641ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1092023-12-05T05:22:43.641ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:346
1102023-12-05T05:22:43.641ZINFOSledAgent (StorageManager): Attempting to add queued disks file = sled-storage/src/manager.rs:394 num_disks = 9
1112023-12-05T05:22:43.644ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3270
1122023-12-05T05:22:43.651ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1132023-12-05T05:22:43.651ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1142023-12-05T05:22:43.651ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1152023-12-05T05:22:43.651ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1162023-12-05T05:22:43.651ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1172023-12-05T05:22:43.651ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1182023-12-05T05:22:43.682ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1192023-12-05T05:22:43.687ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
1202023-12-05T05:22:43.690ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone file = sled-storage/src/dataset.rs:323
1212023-12-05T05:22:43.709ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:305 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
1222023-12-05T05:22:43.709ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1232023-12-05T05:22:43.809ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1242023-12-05T05:22:43.809ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1252023-12-05T05:22:43.809ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1262023-12-05T05:22:43.809ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1272023-12-05T05:22:43.810ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1282023-12-05T05:22:43.810ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1292023-12-05T05:22:43.839ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1302023-12-05T05:22:43.846ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone file = sled-storage/src/dataset.rs:323
1312023-12-05T05:22:43.966ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1322023-12-05T05:22:43.966ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1332023-12-05T05:22:43.966ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1342023-12-05T05:22:43.966ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1352023-12-05T05:22:43.966ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1362023-12-05T05:22:43.966ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1372023-12-05T05:22:43.994ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1382023-12-05T05:22:44.001ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone file = sled-storage/src/dataset.rs:323
1392023-12-05T05:22:44.120ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1402023-12-05T05:22:44.120ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1412023-12-05T05:22:44.120ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1422023-12-05T05:22:44.120ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1432023-12-05T05:22:44.120ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1442023-12-05T05:22:44.120ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1452023-12-05T05:22:44.149ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1462023-12-05T05:22:44.156ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1472023-12-05T05:22:44.280ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1482023-12-05T05:22:44.281ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1492023-12-05T05:22:44.281ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1502023-12-05T05:22:44.281ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1512023-12-05T05:22:44.281ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1522023-12-05T05:22:44.281ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1532023-12-05T05:22:44.308ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1542023-12-05T05:22:44.315ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone file = sled-storage/src/dataset.rs:323
1552023-12-05T05:22:44.433ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1562023-12-05T05:22:44.433ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1572023-12-05T05:22:44.433ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1582023-12-05T05:22:44.433ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1592023-12-05T05:22:44.433ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1602023-12-05T05:22:44.433ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1612023-12-05T05:22:44.460ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1622023-12-05T05:22:44.468ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1632023-12-05T05:22:44.586ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1642023-12-05T05:22:44.586ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1652023-12-05T05:22:44.586ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1662023-12-05T05:22:44.586ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1672023-12-05T05:22:44.586ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1682023-12-05T05:22:44.586ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1692023-12-05T05:22:44.615ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1702023-12-05T05:22:44.623ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone file = sled-storage/src/dataset.rs:323
1712023-12-05T05:22:44.744ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1722023-12-05T05:22:44.744ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1732023-12-05T05:22:44.744ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1742023-12-05T05:22:44.744ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1752023-12-05T05:22:44.744ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1762023-12-05T05:22:44.744ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1772023-12-05T05:22:44.772ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1782023-12-05T05:22:44.780ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1792023-12-05T05:22:44.900ZINFOSledAgent (StorageManager): Loading latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} file = sled-storage/src/dataset.rs:261
1802023-12-05T05:22:44.900ZINFOSledAgent (StorageManager): Loaded latest secret disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:263
1812023-12-05T05:22:44.900ZINFOSledAgent (StorageManager): Retrieving key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:267
1822023-12-05T05:22:44.900ZINFOSledAgent (StorageManager): Got key disk_id = DiskIdentity {\n vendor: "synthetic-vendor",\n serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03",\n model: "synthetic-model",\n} epoch = 0 file = sled-storage/src/dataset.rs:269
1832023-12-05T05:22:44.900ZINFOSledAgent (StorageManager): Created keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:37
1842023-12-05T05:22:44.900ZINFOSledAgent (StorageManager): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:281
1852023-12-05T05:22:44.929ZINFOSledAgent (StorageManager): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03-synthetic-model-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:56
1862023-12-05T05:22:44.936ZINFOSledAgent (StorageManager): Automatically destroying dataset: oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone file = sled-storage/src/dataset.rs:323
1872023-12-05T05:22:45.046ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
1882023-12-05T05:22:45.046ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
1892023-12-05T05:22:45.046ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:317 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
1902023-12-05T05:22:45.052ZINFOSledAgent (StorageMonitor): Received storage manager update file = sled-agent/src/storage_monitor.rs:111 resources = StorageResources { disks: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 2655744, free: 20935309824, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 771072, free: 20937194496, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" }, zpool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }), Pool { name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, info: ZpoolInfo { name: "oxp_31bd71cd-4736-4a12-a387-9b74b050396f", size: 20937965568, allocated: 774144, free: 20937191424, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" }, zpool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } }), Pool { name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, info: ZpoolInfo { name: "oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7", size: 20937965568, allocated: 777216, free: 20937188352, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: a462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 497664, free: 20937467904, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal } }), Pool { name: ZpoolName { id: b462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: Internal }, info: ZpoolInfo { name: "oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 497664, free: 20937467904, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" }, zpool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } }), Pool { name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, info: ZpoolInfo { name: "oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930", size: 20937965568, allocated: 774144, free: 20937191424, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" }, zpool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External } }), Pool { name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External }, info: ZpoolInfo { name: "oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60", size: 20937965568, allocated: 821760, free: 20937143808, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" }, zpool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External } }), Pool { name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External }, info: ZpoolInfo { name: "oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b", size: 20937965568, allocated: 771072, free: 20937194496, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 783360, free: 20937182208, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } }), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }: (Synthetic(SyntheticDisk { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" }, zpool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }), Pool { name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, info: ZpoolInfo { name: "oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03", size: 20937965568, allocated: 778752, free: 20937186816, health: Online }, parent: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } })} }
1912023-12-05T05:22:45.070ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1922023-12-05T05:22:45.072ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:314
1932023-12-05T05:22:45.091ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:353
1942023-12-05T05:22:45.093ZINFOSledAgent: successfully created encrypted zvol: name="oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1952023-12-05T05:22:45.093ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/swap" file = sled-agent/src/swap_device.rs:96
1962023-12-05T05:22:45.098ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:332 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
1972023-12-05T05:22:45.098ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1982023-12-05T05:22:45.098ZINFOSledAgent (BackingFs): Ensuring dataset oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/backing/fmd file = sled-agent/src/backing_fs.rs:136
1992023-12-05T05:22:45.160ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
2002023-12-05T05:22:45.379ZINFOSledAgent (BackingFs): Mounting oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
2012023-12-05T05:22:45.392ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
2022023-12-05T05:22:45.392ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
2032023-12-05T05:22:45.392ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
2042023-12-05T05:22:45.392ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
2052023-12-05T05:22:45.562ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:162
2062023-12-05T05:22:46.087ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:80 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2072023-12-05T05:22:46.087ZINFOSledAgent (StorageMonitor): Underlay Available file = sled-agent/src/storage_monitor.rs:122 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2082023-12-05T05:22:46.087ZINFOSledAgent (InstanceManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/instance_manager.rs:177
2092023-12-05T05:22:46.087ZINFOSledAgent (InstanceManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/instance_manager.rs:185
2102023-12-05T05:22:46.100ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: M2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2112023-12-05T05:22:46.101ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/616b26df-e62a-4c68-b506-f4a923d8aaf7): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2122023-12-05T05:22:46.101ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/a462a7f7-b628-40fe-80ff-4e4189e2d62b): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2132023-12-05T05:22:46.101ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/d462a7f7-b628-40fe-80ff-4e4189e2d62b): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2142023-12-05T05:22:46.101ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2152023-12-05T05:22:46.101ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2162023-12-05T05:22:46.101ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2172023-12-05T05:22:46.101ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2182023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: M2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2192023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2202023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2212023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2222023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2232023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/physical-disk): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:219
2242023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/14b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2252023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/24b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2262023-12-05T05:22:46.102ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_31bd71cd-4736-4a12-a387-9b74b050396f on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/31bd71cd-4736-4a12-a387-9b74b050396f): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2272023-12-05T05:22:46.103ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/b462a7f7-b628-40fe-80ff-4e4189e2d62b): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2282023-12-05T05:22:46.103ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/cd70d7f6-2354-4bf2-8012-55bf9eaf7930): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2292023-12-05T05:22:46.103ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/ceb4461c-cf56-4719-ad3c-14430bfdfb60): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2302023-12-05T05:22:46.103ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/e4b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2312023-12-05T05:22:46.103ZINFOSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384/zpools/f4b4dc87-ab46-49fb-a4b4-d361ae214c03): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/storage_monitor.rs:270
2322023-12-05T05:22:48.316ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_switch
2332023-12-05T05:22:48.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 44.601115ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2342023-12-05T05:22:48.688ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
2352023-12-05T05:22:48.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 92.837631ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2362023-12-05T05:22:49.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 103.879146ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2372023-12-05T05:22:49.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 535.045534ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2382023-12-05T05:22:49.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 852.429144ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2392023-12-05T05:22:50.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.11166828s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2402023-12-05T05:22:52.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.471494777s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2412023-12-05T05:22:53.689ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
2422023-12-05T05:22:54.805ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:1716
2432023-12-05T05:22:55.954ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:938 underlay_address = fd00:1122:3344:101::1
2442023-12-05T05:22:55.954ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2452023-12-05T05:22:56.012ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::1]:12345
2462023-12-05T05:22:56.012ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:424
2472023-12-05T05:22:56.012ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2482023-12-05T05:22:56.012ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2492023-12-05T05:22:56.012ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:198
2502023-12-05T05:22:56.013ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:198
2512023-12-05T05:22:56.013ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 309.146874ms sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2522023-12-05T05:22:56.013ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:137
2532023-12-05T05:22:56.013ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:241 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:1::, prefix: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Network { addr: 0.0.0.0, prefix: 0 }), nexthop: 10.151.1.100 }], addresses: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 32 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [] })))
2542023-12-05T05:22:56.013ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3042
2552023-12-05T05:22:56.013ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:56047
2562023-12-05T05:22:56.016ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2572023-12-05T05:22:56.016ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2582023-12-05T05:22:56.016ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2592023-12-05T05:22:56.016ZWARNSledAgent (RSS): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json file = common/src/ledger.rs:187
2602023-12-05T05:22:56.016ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json
2612023-12-05T05:22:56.016ZWARNSledAgent (RSS): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json file = common/src/ledger.rs:187
2622023-12-05T05:22:56.016ZERROSledAgent (RSS): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/rss-service-plan-v2.json
2632023-12-05T05:22:56.016ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2642023-12-05T05:22:56.017ZINFOSledAgent: failed to notify nexus about sled agent error = Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/sled_agent.rs:666 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2652023-12-05T05:22:56.049ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59797
2662023-12-05T05:22:56.049ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2672023-12-05T05:22:56.049ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 181 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59797 req_id = 5139b6e4-0067-485c-bb82-f7ad80c641b0 response_code = 200 uri = /zpools
2682023-12-05T05:22:56.079ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36172
2692023-12-05T05:22:56.079ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36172 req_id = 143112dc-ee63-4d06-8f19-54daabdf9e92 response_code = 200 uri = /sled-role
2702023-12-05T05:22:56.081ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2712023-12-05T05:22:56.081ZINFOSledAgent (RSS): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-service-plan-v2.json.tmp file = common/src/ledger.rs:198
2722023-12-05T05:22:56.081ZINFOSledAgent (RSS): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-service-plan-v2.json.tmp file = common/src/ledger.rs:198
2732023-12-05T05:22:56.082ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:749
2742023-12-05T05:22:56.114ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: bd82653d-a25b-409a-af88-7344f9f8a41b, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: b96c752c-8512-47e8-bbf5-dddb77760297, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
2752023-12-05T05:22:56.114ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58957
2762023-12-05T05:22:56.114ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
2772023-12-05T05:22:56.114ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
2782023-12-05T05:22:56.134ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2792023-12-05T05:22:56.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
2802023-12-05T05:22:56.167ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
2812023-12-05T05:22:56.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
2822023-12-05T05:22:56.216ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
2832023-12-05T05:22:56.238ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2842023-12-05T05:22:56.264ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2852023-12-05T05:22:56.264ZWARNSledAgent (ServiceManager): No ledger in /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:187
2862023-12-05T05:22:56.264ZERROSledAgent (ServiceManager): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json
2872023-12-05T05:22:56.264ZWARNSledAgent (ServiceManager): No ledger in /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:187
2882023-12-05T05:22:56.264ZERROSledAgent (ServiceManager): Failed to read ledger: Not found in storage file = common/src/ledger.rs:91 path = /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json
2892023-12-05T05:22:56.264ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2902023-12-05T05:22:56.267ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2912023-12-05T05:22:56.268ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2922023-12-05T05:22:56.269ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
2932023-12-05T05:22:56.322ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b file = illumos-utils/src/zone.rs:314
2942023-12-05T05:22:56.326ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 579.16956ms sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2952023-12-05T05:22:56.327ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297 file = illumos-utils/src/zone.rs:314
2962023-12-05T05:22:56.331ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc file = illumos-utils/src/zone.rs:314
2972023-12-05T05:22:56.332ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
2982023-12-05T05:22:56.344ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b file = illumos-utils/src/zone.rs:353
2992023-12-05T05:22:56.348ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297 file = illumos-utils/src/zone.rs:353
3002023-12-05T05:22:56.352ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc file = illumos-utils/src/zone.rs:353
3012023-12-05T05:22:56.455ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3022023-12-05T05:22:56.629ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3032023-12-05T05:22:56.817ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3042023-12-05T05:22:56.907ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 1.197662391s sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3052023-12-05T05:22:57.048ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3062023-12-05T05:22:57.381ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3072023-12-05T05:22:57.769ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3082023-12-05T05:22:57.998ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3092023-12-05T05:22:58.053ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3102023-12-05T05:22:58.057ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3112023-12-05T05:22:58.106ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 2.069656558s sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3122023-12-05T05:22:58.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 39.525238ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3132023-12-05T05:22:58.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 71.064862ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3142023-12-05T05:22:58.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 33.665395ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3152023-12-05T05:22:58.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 99.034421ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3162023-12-05T05:22:58.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 118.198796ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3172023-12-05T05:22:58.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 85.619183ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3182023-12-05T05:22:58.365ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3192023-12-05T05:22:58.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 247.746552ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3202023-12-05T05:22:58.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 162.47205ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3212023-12-05T05:22:58.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 169.765355ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3222023-12-05T05:22:58.692ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
3232023-12-05T05:22:58.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 536.77467ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3242023-12-05T05:22:58.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 275.580067ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3252023-12-05T05:22:58.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 371.592104ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3262023-12-05T05:22:59.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 537.207007ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3272023-12-05T05:22:59.192ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3282023-12-05T05:22:59.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 869.47983ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3292023-12-05T05:22:59.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 498.330977ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3302023-12-05T05:22:59.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 1.258973042s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3312023-12-05T05:22:59.761ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:9fb2::2 exists in switch zone file = sled-agent/src/services.rs:1733
3322023-12-05T05:22:59.761ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:625 zone = oxz_switch
3332023-12-05T05:22:59.764ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3342023-12-05T05:22:59.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 1.012519005s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3352023-12-05T05:23:00.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 619.718335ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3362023-12-05T05:23:00.178ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 2.090157559s sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3372023-12-05T05:23:00.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 770.333609ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3382023-12-05T05:23:00.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 516.086442ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3392023-12-05T05:23:00.890ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3402023-12-05T05:23:00.909ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 1.317589281s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3412023-12-05T05:23:01.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 939.147618ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3422023-12-05T05:23:01.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 729.107937ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3432023-12-05T05:23:01.870ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3442023-12-05T05:23:02.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b") failed: Property not found. retry in 833.947729ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3452023-12-05T05:23:02.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297") failed: Property not found. retry in 878.361482ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3462023-12-05T05:23:02.271ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1060 retry_after = 10.390255131s sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3472023-12-05T05:23:02.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc") failed: Property not found. retry in 1.317674276s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3482023-12-05T05:23:03.114ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:1::1 exists file = sled-agent/src/services.rs:1769
3492023-12-05T05:23:03.114ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:1::1, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_internal_dns_bd82653d-a25b-409a-af88-7344f9f8a41b
3502023-12-05T05:23:03.679ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:1::1 exists - OK file = sled-agent/src/services.rs:1777
3512023-12-05T05:23:03.679ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3522023-12-05T05:23:03.905ZINFOSledAgent (ServiceManager): Setting up internal-dns service file = sled-agent/src/services.rs:1995
3532023-12-05T05:23:04.291ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fe8f:33f8 file = sled-agent/src/services.rs:1740
3542023-12-05T05:23:04.296ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3552023-12-05T05:23:04.416ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2280
3562023-12-05T05:23:04.431ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
3572023-12-05T05:23:04.431ZINFOSledAgent: contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384 sled_id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
3582023-12-05T05:23:04.436ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {}
3592023-12-05T05:23:04.486ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:2::1 exists file = sled-agent/src/services.rs:1769
3602023-12-05T05:23:04.486ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:2::1, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297
3612023-12-05T05:23:04.674ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2159
3622023-12-05T05:23:04.964ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2194
3632023-12-05T05:23:05.034ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:2::1 exists - OK file = sled-agent/src/services.rs:1777
3642023-12-05T05:23:05.034ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3652023-12-05T05:23:05.159ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2461
3662023-12-05T05:23:05.210ZINFOSledAgent (ServiceManager): Setting up internal-dns service file = sled-agent/src/services.rs:1995
3672023-12-05T05:23:05.302ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2466
3682023-12-05T05:23:05.483ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2400
3692023-12-05T05:23:05.646ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2188
3702023-12-05T05:23:05.656ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3294 new = [fd00:1122:3344:101::2, ::1] old = [::1]
3712023-12-05T05:23:05.656ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3310
3722023-12-05T05:23:05.656ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_switch
3732023-12-05T05:23:05.765ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:3::1 exists file = sled-agent/src/services.rs:1769
3742023-12-05T05:23:05.765ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:3::1, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_internal_dns_44ac5ff3-cccf-4698-a3c5-2fe40573f2cc
3752023-12-05T05:23:06.206ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3318
3762023-12-05T05:23:06.211ZINFOSledAgent (ServiceManager): configuring dendrite zone file = sled-agent/src/services.rs:3374
3772023-12-05T05:23:06.309ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:3::1 exists - OK file = sled-agent/src/services.rs:1777
3782023-12-05T05:23:06.309ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:1801
3792023-12-05T05:23:06.398ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3414 rack_subnet = fd00:1122:3344::
3802023-12-05T05:23:06.450ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:359
3812023-12-05T05:23:06.483ZINFOSledAgent (ServiceManager): Setting up internal-dns service file = sled-agent/src/services.rs:1995
3822023-12-05T05:23:06.486ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 51.062265ms) error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "a9aa6716-78f3-45e4-a07d-29984e237c27", "content-length": "152", "date": "Tue, 05 Dec 2023 05:23:06 GMT"}; value: Error { error_code: Some("DiscoveryNotYetComplete"), message: "discovery process not yet complete", request_id: "a9aa6716-78f3-45e4-a07d-29984e237c27" } file = sled-agent/src/bootstrap/early_networking.rs:378
3832023-12-05T05:23:06.538ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 139.424718ms) error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "889fe226-71f6-4f4f-b68e-8f90fbd0f06e", "content-length": "152", "date": "Tue, 05 Dec 2023 05:23:06 GMT"}; value: Error { error_code: Some("DiscoveryNotYetComplete"), message: "discovery process not yet complete", request_id: "889fe226-71f6-4f4f-b68e-8f90fbd0f06e" } file = sled-agent/src/bootstrap/early_networking.rs:378
3842023-12-05T05:23:06.678ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 218.459935ms) error = Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "7b99ebcc-3f61-4ae9-ab34-8272615b059b", "content-length": "152", "date": "Tue, 05 Dec 2023 05:23:06 GMT"}; value: Error { error_code: Some("DiscoveryNotYetComplete"), message: "discovery process not yet complete", request_id: "7b99ebcc-3f61-4ae9-ab34-8272615b059b" } file = sled-agent/src/bootstrap/early_networking.rs:378
3852023-12-05T05:23:06.898ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:407
3862023-12-05T05:23:06.930ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:533
3872023-12-05T05:23:06.930ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:602
3882023-12-05T05:23:06.930ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:605 uptime = 2
3892023-12-05T05:23:06.930ZINFOSledAgent (ServiceManager): Configuring boundary services loopback address on switch config = Ipv6Entry {\n addr: fd00:99::1,\n tag: "omicron",\n} file = sled-agent/src/bootstrap/early_networking.rs:429
3902023-12-05T05:23:06.931ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n v4_routes: {\n "0.0.0.0/0": [\n RouteSettingsV4 {\n link_id: 0,\n nexthop: 10.151.1.100,\n },\n ],\n },\n v6_routes: {},\n} file = sled-agent/src/bootstrap/early_networking.rs:440
3912023-12-05T05:23:06.931ZINFOSledAgent (ServiceManager): advertising boundary services loopback address file = sled-agent/src/bootstrap/early_networking.rs:457
3922023-12-05T05:23:06.960ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
3932023-12-05T05:23:07.014ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
3942023-12-05T05:23:07.014ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
3952023-12-05T05:23:07.014ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
3962023-12-05T05:23:07.014ZINFOSledAgent (ServiceManager): Sending prefix to ddmd for advertisement DdmAdminClient = [fd00:1122:3344:101::2]:8000 file = clients/ddm-admin-client/src/lib.rs:91 prefix = Ipv6Prefix { addr: fd00:99::, len: 64 }
3972023-12-05T05:23:07.015ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 10900414 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:58957 req_id = 10d5adb3-c466-4392-8624-f0f8dbb5a76e response_code = 204 uri = /omicron-zones
3982023-12-05T05:23:07.017ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:417
3992023-12-05T05:23:07.060ZINFOSledAgent (StorageManager): Received DisksChanged({}) file = sled-storage/src/manager.rs:346
4002023-12-05T05:23:07.079ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:417
4012023-12-05T05:23:07.119ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:417
4022023-12-05T05:23:07.168ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:453
4032023-12-05T05:23:07.168ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
4042023-12-05T05:23:07.199ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
4052023-12-05T05:23:07.200ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
4062023-12-05T05:23:07.200ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
4072023-12-05T05:23:07.229ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: bd82653d-a25b-409a-af88-7344f9f8a41b, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: b96c752c-8512-47e8-bbf5-dddb77760297, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: fa05c9b8-e2a1-4c10-a933-8980828c20ab, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 7fe67b7a-7621-4dfb-a4f7-62a16105eaa7, kind: Service { id: fa05c9b8-e2a1-4c10-a933-8980828c20ab }, name: Name("ntp-fa05c9b8-e2a1-4c10-a933-8980828c20ab"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
4082023-12-05T05:23:07.229ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65499
4092023-12-05T05:23:07.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
4102023-12-05T05:23:07.230ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
4112023-12-05T05:23:07.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
4122023-12-05T05:23:07.253ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
4132023-12-05T05:23:07.273ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
4142023-12-05T05:23:07.273ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
4152023-12-05T05:23:07.291ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
4162023-12-05T05:23:07.291ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
4172023-12-05T05:23:07.291ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
4182023-12-05T05:23:07.291ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
4192023-12-05T05:23:07.293ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
4202023-12-05T05:23:07.326ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
4212023-12-05T05:23:07.327ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
4222023-12-05T05:23:07.327ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
4232023-12-05T05:23:07.371ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte0", _ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 160, 167]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
4242023-12-05T05:23:07.371ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = ntp
4252023-12-05T05:23:07.372ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
4262023-12-05T05:23:07.372ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
4272023-12-05T05:23:07.372ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
4282023-12-05T05:23:07.373ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
4292023-12-05T05:23:07.396ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab file = illumos-utils/src/zone.rs:314
4302023-12-05T05:23:07.414ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab file = illumos-utils/src/zone.rs:353
4312023-12-05T05:23:08.405ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4322023-12-05T05:23:08.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab") failed: Property not found. retry in 29.688039ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4332023-12-05T05:23:08.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab") failed: Property not found. retry in 77.823813ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4342023-12-05T05:23:08.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab") failed: Property not found. retry in 130.930697ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4352023-12-05T05:23:08.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab") failed: Property not found. retry in 254.333431ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4362023-12-05T05:23:09.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab") failed: Property not found. retry in 1.079587537s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4372023-12-05T05:23:09.438ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
4382023-12-05T05:23:10.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab") failed: Property not found. retry in 1.494002347s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4392023-12-05T05:23:10.253ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64189
4402023-12-05T05:23:10.253ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 95 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64189 req_id = c638b550-f58e-4796-add4-f980a609ff0c response_code = 200 uri = /baseboard
4412023-12-05T05:23:11.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab") failed: Property not found. retry in 1.217808316s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4422023-12-05T05:23:12.938ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::1b exists file = sled-agent/src/services.rs:1769
4432023-12-05T05:23:12.938ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::1b, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4442023-12-05T05:23:13.494ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::1b exists - OK file = sled-agent/src/services.rs:1777
4452023-12-05T05:23:13.494ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
4462023-12-05T05:23:13.696ZINFOSledAgent (ServiceManager): Set up NTP service boundary=true, Servers=["0.pool.ntp.org"] file = sled-agent/src/services.rs:2086
4472023-12-05T05:23:13.723ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab
4482023-12-05T05:23:18.877ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
4492023-12-05T05:23:18.928ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
4502023-12-05T05:23:18.929ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
4512023-12-05T05:23:18.929ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 11699851 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:65499 req_id = 9edb29f7-34b6-49f5-87f3-217bc2c37255 response_code = 204 uri = /omicron-zones
4522023-12-05T05:23:18.931ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:494
4532023-12-05T05:23:18.960ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4542023-12-05T05:23:18.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43990
4552023-12-05T05:23:18.970ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_fa05c9b8-e2a1-4c10-a933-8980828c20ab': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3028
4562023-12-05T05:23:18.970ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 10106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43990 req_id = c463e6b5-0475-4d3b-8d63-096aa860801e response_code = 500 uri = /timesync
4572023-12-05T05:23:18.973ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4582023-12-05T05:23:19.152ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4592023-12-05T05:23:19.152ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38546
4602023-12-05T05:23:19.161ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8844 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38546 req_id = c463552f-10c1-4333-95fb-37b00df2ba56 response_code = 200 uri = /timesync
4612023-12-05T05:23:19.164ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4622023-12-05T05:23:19.164ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4632023-12-05T05:23:19.353ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4642023-12-05T05:23:19.353ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47844
4652023-12-05T05:23:19.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7721 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47844 req_id = 0236eb53-4981-4ae5-b6f3-cf18ff60f797 response_code = 200 uri = /timesync
4662023-12-05T05:23:19.364ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4672023-12-05T05:23:19.364ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4682023-12-05T05:23:19.485ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4692023-12-05T05:23:19.485ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36333
4702023-12-05T05:23:19.494ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36333 req_id = 04ee92bb-28ac-4525-a9ad-b055683a9c00 response_code = 200 uri = /timesync
4712023-12-05T05:23:19.496ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4722023-12-05T05:23:19.496ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4732023-12-05T05:23:19.629ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4742023-12-05T05:23:19.629ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64984
4752023-12-05T05:23:19.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7682 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64984 req_id = e0984969-b0c2-4f91-871c-8bc65a0d5958 response_code = 200 uri = /timesync
4762023-12-05T05:23:19.640ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4772023-12-05T05:23:19.640ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4782023-12-05T05:23:19.854ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4792023-12-05T05:23:19.854ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37407
4802023-12-05T05:23:19.862ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7792 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37407 req_id = 74bcecb6-5ad6-43d2-ac30-a263299347fa response_code = 200 uri = /timesync
4812023-12-05T05:23:19.864ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4822023-12-05T05:23:19.864ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4832023-12-05T05:23:20.031ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4842023-12-05T05:23:20.032ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59344
4852023-12-05T05:23:20.040ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7950 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59344 req_id = f290efa8-9cae-4046-88a0-b246435961f1 response_code = 200 uri = /timesync
4862023-12-05T05:23:20.044ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4872023-12-05T05:23:20.044ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4882023-12-05T05:23:20.514ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4892023-12-05T05:23:20.514ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35255
4902023-12-05T05:23:20.522ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7751 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35255 req_id = c99f0f89-5262-44ff-aa15-6f912bd1b838 response_code = 200 uri = /timesync
4912023-12-05T05:23:20.524ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4922023-12-05T05:23:20.524ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4932023-12-05T05:23:20.889ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4942023-12-05T05:23:20.889ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51502
4952023-12-05T05:23:20.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51502 req_id = f826b85c-bb2b-4f77-a3a6-f7f9c6110ae4 response_code = 200 uri = /timesync
4962023-12-05T05:23:20.901ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
4972023-12-05T05:23:20.901ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
4982023-12-05T05:23:21.471ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
4992023-12-05T05:23:21.471ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44749
5002023-12-05T05:23:21.479ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 7804 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44749 req_id = efdd9381-49c6-46c8-b86c-28a12b85447c response_code = 200 uri = /timesync
5012023-12-05T05:23:21.481ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5022023-12-05T05:23:21.481ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5032023-12-05T05:23:21.969ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5042023-12-05T05:23:21.969ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:32994
5052023-12-05T05:23:21.977ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8010 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:32994 req_id = 46c853e2-4f5a-40e9-b1ca-bde8ddb2dfc8 response_code = 200 uri = /timesync
5062023-12-05T05:23:21.979ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5072023-12-05T05:23:21.979ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5082023-12-05T05:23:22.426ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5092023-12-05T05:23:22.427ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61586
5102023-12-05T05:23:22.436ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8784 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61586 req_id = 5ffb3709-eb05-4afa-84cd-b791036f0610 response_code = 200 uri = /timesync
5112023-12-05T05:23:22.439ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5122023-12-05T05:23:22.439ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5132023-12-05T05:23:23.397ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5142023-12-05T05:23:23.397ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40281
5152023-12-05T05:23:23.406ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8877 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40281 req_id = c0116bf7-f09e-47c6-8088-2586c090544a response_code = 200 uri = /timesync
5162023-12-05T05:23:23.409ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5172023-12-05T05:23:23.409ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5182023-12-05T05:23:23.881ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
5192023-12-05T05:23:24.721ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5202023-12-05T05:23:24.721ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54838
5212023-12-05T05:23:24.725ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
5222023-12-05T05:23:24.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 9044 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54838 req_id = 067d1159-7fd7-4f50-90ba-9eb0249dd5de response_code = 200 uri = /timesync
5232023-12-05T05:23:24.733ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1701753797.8763824, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5242023-12-05T05:23:24.733ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:524
5252023-12-05T05:23:25.586ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:474
5262023-12-05T05:23:25.586ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42794
5272023-12-05T05:23:25.590ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
5282023-12-05T05:23:25.606ZWARNSledAgent (ServiceManager): Updating [wu]tmpx databases failed: Command [/usr/platform/oxide/bin/tmpx -Z] executed and failed with status: exit status: 1 stdout: [zone:global:0] Boot time: 1701753339 Root path: Updating database '/var/adm/wtmpx' First entry looks correct; nothing to do Updating database '/var/adm/utmpx' First entry looks correct; nothing to do [zone:sidecar_softnpu:1] Boot time: 1701753757 Root path: /sidecar/sidecar_softnpu/root Updating database '/sidecar/sidecar_softnpu/root/var/adm/wtmpx' First entry looks correct; nothing to do Updating database '/sidecar/sidecar_softnpu/root/var/adm/utmpx' First entry looks correct; nothing to do [zone:oxz_switch:2] Boot time: 1701753768 Root path: /zone/oxz_switch/root Updating database '/zone/oxz_switch/root/var/adm/wtmpx' First entry looks correct; nothing to do Updating database '/zone/oxz_switch/root/var/adm/utmpx' First entry looks correct; nothing to do [zone:oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297:3] Boot time: 1701753778 Root path: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297/root Updating database '/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297/root/var/adm/wtmpx' stderr: tmpx: Invalid database: '/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b96c752c-8512-47e8-bbf5-dddb77760297/root/var/adm/wtmpx' (see utmpxname(3C)) file = sled-agent/src/services.rs:2949
5292023-12-05T05:23:25.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 19696 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42794 req_id = 4c0ee15f-1f58-482c-bab2-4f33170964c4 response_code = 200 uri = /timesync
5302023-12-05T05:23:25.610ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 3423718683, ip_addr: 204.17.205.27, stratum: 3, ref_time: 1701753805.1269531, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:502
5312023-12-05T05:23:25.610ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1042
5322023-12-05T05:23:25.641ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd82653d-a25b-409a-af88-7344f9f8a41b, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: b96c752c-8512-47e8-bbf5-dddb77760297, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: fa05c9b8-e2a1-4c10-a933-8980828c20ab, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 7fe67b7a-7621-4dfb-a4f7-62a16105eaa7, kind: Service { id: fa05c9b8-e2a1-4c10-a933-8980828c20ab }, name: Name("ntp-fa05c9b8-e2a1-4c10-a933-8980828c20ab"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 90d6cc55-c535-4b71-a7d5-769e23015de9, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 32271630-ccab-448f-a627-a273a66ed73f, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 876fd524-b8a9-4c79-a2ea-fcf06eb6279e, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } } } }, OmicronZoneConfig { id: a48280e1-9a76-43b1-b5d2-6ba2da05e4d5, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } } } }, OmicronZoneConfig { id: fa06112d-8305-4f51-bbf3-54d846151c2a, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
5332023-12-05T05:23:25.641ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58699
5342023-12-05T05:23:25.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5352023-12-05T05:23:25.642ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5362023-12-05T05:23:25.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5372023-12-05T05:23:25.662ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5382023-12-05T05:23:25.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5392023-12-05T05:23:25.680ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5402023-12-05T05:23:25.699ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 90d6cc55-c535-4b71-a7d5-769e23015de9, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5412023-12-05T05:23:25.699ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 90d6cc55-c535-4b71-a7d5-769e23015de9, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5422023-12-05T05:23:25.746ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32271630-ccab-448f-a627-a273a66ed73f, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5432023-12-05T05:23:25.746ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 32271630-ccab-448f-a627-a273a66ed73f, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5442023-12-05T05:23:26.018ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
5452023-12-05T05:23:26.037ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 876fd524-b8a9-4c79-a2ea-fcf06eb6279e, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5462023-12-05T05:23:26.037ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 876fd524-b8a9-4c79-a2ea-fcf06eb6279e, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5472023-12-05T05:23:26.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a48280e1-9a76-43b1-b5d2-6ba2da05e4d5, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5482023-12-05T05:23:26.086ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: a48280e1-9a76-43b1-b5d2-6ba2da05e4d5, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5492023-12-05T05:23:26.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fa06112d-8305-4f51-bbf3-54d846151c2a, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
5502023-12-05T05:23:26.134ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: fa06112d-8305-4f51-bbf3-54d846151c2a, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
5512023-12-05T05:23:26.184ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5522023-12-05T05:23:26.184ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
5532023-12-05T05:23:26.185ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
5542023-12-05T05:23:26.185ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5552023-12-05T05:23:26.192ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5562023-12-05T05:23:26.192ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5572023-12-05T05:23:26.192ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5582023-12-05T05:23:26.192ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5592023-12-05T05:23:26.192ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
5602023-12-05T05:23:26.285ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5 file = illumos-utils/src/zone.rs:314
5612023-12-05T05:23:26.289ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f file = illumos-utils/src/zone.rs:314
5622023-12-05T05:23:26.294ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9 file = illumos-utils/src/zone.rs:314
5632023-12-05T05:23:26.298ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a file = illumos-utils/src/zone.rs:314
5642023-12-05T05:23:26.303ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e file = illumos-utils/src/zone.rs:314
5652023-12-05T05:23:26.307ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5 file = illumos-utils/src/zone.rs:353
5662023-12-05T05:23:26.312ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f file = illumos-utils/src/zone.rs:353
5672023-12-05T05:23:26.316ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9 file = illumos-utils/src/zone.rs:353
5682023-12-05T05:23:26.320ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a file = illumos-utils/src/zone.rs:353
5692023-12-05T05:23:26.324ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e file = illumos-utils/src/zone.rs:353
5702023-12-05T05:23:28.647ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
5712023-12-05T05:23:30.305ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::5"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5722023-12-05T05:23:30.305ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
5732023-12-05T05:23:30.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 27.230001ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
5742023-12-05T05:23:30.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 136.392237ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
5752023-12-05T05:23:30.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 271.568239ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
5762023-12-05T05:23:30.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 363.447959ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
5772023-12-05T05:23:31.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 780.155937ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
5782023-12-05T05:23:31.673ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::3"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5792023-12-05T05:23:31.673ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
5802023-12-05T05:23:31.681ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::4"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5812023-12-05T05:23:31.681ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
5822023-12-05T05:23:31.762ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::6"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5832023-12-05T05:23:31.762ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
5842023-12-05T05:23:31.771ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::7"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5852023-12-05T05:23:31.772ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
5862023-12-05T05:23:31.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 55.687059ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
5872023-12-05T05:23:31.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 56.611874ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
5882023-12-05T05:23:31.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 149.044193ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
5892023-12-05T05:23:31.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 82.517557ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
5902023-12-05T05:23:32.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 177.270458ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
5912023-12-05T05:23:32.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 253.879607ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
5922023-12-05T05:23:32.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 1.272412152s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
5932023-12-05T05:23:32.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 27.039846ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
5942023-12-05T05:23:32.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 61.223036ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
5952023-12-05T05:23:32.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 117.657366ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
5962023-12-05T05:23:32.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 114.684864ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
5972023-12-05T05:23:32.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 223.041949ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
5982023-12-05T05:23:32.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 150.679298ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
5992023-12-05T05:23:32.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 256.451397ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6002023-12-05T05:23:32.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 548.311089ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
6012023-12-05T05:23:32.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 559.075265ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
6022023-12-05T05:23:32.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 207.388685ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6032023-12-05T05:23:32.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 357.583762ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6042023-12-05T05:23:32.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 751.771169ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6052023-12-05T05:23:32.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 476.919356ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
6062023-12-05T05:23:33.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 806.842602ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6072023-12-05T05:23:33.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 922.42928ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
6082023-12-05T05:23:33.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 725.445124ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
6092023-12-05T05:23:33.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 1.245715472s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
6102023-12-05T05:23:33.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 785.81414ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6112023-12-05T05:23:33.649ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6122023-12-05T05:23:33.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 1.118953433s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6132023-12-05T05:23:34.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 1.392297423s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
6142023-12-05T05:23:34.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 1.133775134s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
6152023-12-05T05:23:34.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 520.090658ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6162023-12-05T05:23:34.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 1.392099317s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
6172023-12-05T05:23:34.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 1.144924459s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6182023-12-05T05:23:34.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 636.719368ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6192023-12-05T05:23:35.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e") failed: Property not found. retry in 1.120853962s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_876fd524-b8a9-4c79-a2ea-fcf06eb6279e
6202023-12-05T05:23:35.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 520.166409ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
6212023-12-05T05:23:35.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 863.954821ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6222023-12-05T05:23:35.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 1.32405748s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
6232023-12-05T05:23:35.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 920.625526ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6242023-12-05T05:23:36.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 536.713081ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
6252023-12-05T05:23:36.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 784.359889ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6262023-12-05T05:23:36.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 871.871767ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
6272023-12-05T05:23:36.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 839.801067ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6282023-12-05T05:23:37.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 1.349138402s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
6292023-12-05T05:23:37.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 1.051751365s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6302023-12-05T05:23:37.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f") failed: Property not found. retry in 823.863194ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f
6312023-12-05T05:23:37.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 863.376261ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6322023-12-05T05:23:38.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a") failed: Property not found. retry in 1.070621651s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fa06112d-8305-4f51-bbf3-54d846151c2a
6332023-12-05T05:23:38.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5") failed: Property not found. retry in 1.295651705s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a48280e1-9a76-43b1-b5d2-6ba2da05e4d5
6342023-12-05T05:23:38.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9") failed: Property not found. retry in 1.386147854s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_90d6cc55-c535-4b71-a7d5-769e23015de9
6352023-12-05T05:23:38.650ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6362023-12-05T05:23:41.602ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58002
6372023-12-05T05:23:41.606ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 438 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58002 req_id = 029f3d91-f033-41c6-bccb-a24439aae1b9 response_code = 200 uri = /baseboard
6382023-12-05T05:23:41.637ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
6392023-12-05T05:23:41.637ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
6402023-12-05T05:23:41.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 15752882 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:58699 req_id = cccb59ca-4039-4a05-8e8c-ead17a5c48e2 response_code = 204 uri = /omicron-zones
6412023-12-05T05:23:41.671ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40426
6422023-12-05T05:23:41.701ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 29968 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = 8eb00169-3c67-408e-8ca8-0beeeebd726e response_code = 500 uri = /cockroachdb
6432023-12-05T05:23:41.701ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8eb00169-3c67-408e-8ca8-0beeeebd726e", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8eb00169-3c67-408e-8ca8-0beeeebd726e" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 145.845941ms
6442023-12-05T05:23:41.879ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 30389 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = 00a82451-d8dd-4a9b-8cad-8e2a2bc10faa response_code = 500 uri = /cockroachdb
6452023-12-05T05:23:41.879ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "00a82451-d8dd-4a9b-8cad-8e2a2bc10faa", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "00a82451-d8dd-4a9b-8cad-8e2a2bc10faa" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 81.901006ms
6462023-12-05T05:23:41.993ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 30240 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = 44c91044-f98b-4f4c-8851-2efbc58f8d63 response_code = 500 uri = /cockroachdb
6472023-12-05T05:23:41.993ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "44c91044-f98b-4f4c-8851-2efbc58f8d63", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "44c91044-f98b-4f4c-8851-2efbc58f8d63" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 191.825374ms
6482023-12-05T05:23:42.214ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 28026 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = a73c5154-0204-4911-be31-366a36b905e2 response_code = 500 uri = /cockroachdb
6492023-12-05T05:23:42.214ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a73c5154-0204-4911-be31-366a36b905e2", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a73c5154-0204-4911-be31-366a36b905e2" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 141.550708ms
6502023-12-05T05:23:42.384ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 27599 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = 218c6d6d-da7a-4f99-a786-6afd12f0b6bd response_code = 500 uri = /cockroachdb
6512023-12-05T05:23:42.385ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "218c6d6d-da7a-4f99-a786-6afd12f0b6bd", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "218c6d6d-da7a-4f99-a786-6afd12f0b6bd" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 210.646185ms
6522023-12-05T05:23:42.624ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 26875 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = b065a70f-03c3-4e79-bb4c-17ea6f9ee5fb response_code = 500 uri = /cockroachdb
6532023-12-05T05:23:42.624ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b065a70f-03c3-4e79-bb4c-17ea6f9ee5fb", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b065a70f-03c3-4e79-bb4c-17ea6f9ee5fb" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 180.334935ms
6542023-12-05T05:23:42.833ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 26418 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = dae98fa0-eb26-4f36-80dd-1007552d46af response_code = 500 uri = /cockroachdb
6552023-12-05T05:23:42.833ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dae98fa0-eb26-4f36-80dd-1007552d46af", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dae98fa0-eb26-4f36-80dd-1007552d46af" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 421.63511ms
6562023-12-05T05:23:43.284ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 27525 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = fde3000b-988d-4041-8ed4-1da1ea74fabc response_code = 500 uri = /cockroachdb
6572023-12-05T05:23:43.284ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fde3000b-988d-4041-8ed4-1da1ea74fabc", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fde3000b-988d-4041-8ed4-1da1ea74fabc" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 335.378173ms
6582023-12-05T05:23:43.647ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 25755 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = b846fff2-162b-4d2b-b025-e63885ae121e response_code = 500 uri = /cockroachdb
6592023-12-05T05:23:43.647ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b846fff2-162b-4d2b-b025-e63885ae121e", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b846fff2-162b-4d2b-b025-e63885ae121e" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 617.841968ms
6602023-12-05T05:23:43.651ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6612023-12-05T05:23:44.294ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 27464 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = 2dc87e01-de00-4af3-a155-93c9d7086e8a response_code = 500 uri = /cockroachdb
6622023-12-05T05:23:44.294ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2dc87e01-de00-4af3-a155-93c9d7086e8a", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2dc87e01-de00-4af3-a155-93c9d7086e8a" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 369.407681ms
6632023-12-05T05:23:44.693ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService6/omicron6 in oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_32271630-ccab-448f-a627-a273a66ed73f /usr/sbin/ipadm show-addr -p -o ADDR oxControlService6/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841 latency_us = 28719 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = f67b44ca-fca2-44ac-a3e5-20f0728e30f7 response_code = 500 uri = /cockroachdb
6642023-12-05T05:23:44.693ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f67b44ca-fca2-44ac-a3e5-20f0728e30f7", "content-length": "124", "date": "Tue, 05 Dec 2023 05:23:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f67b44ca-fca2-44ac-a3e5-20f0728e30f7" } file = sled-agent/src/rack_setup/service.rs:758 retry_after = 773.369948ms
6652023-12-05T05:23:45.497ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::4]:32221 file = sled-agent/src/services.rs:2881
6662023-12-05T05:23:59.244ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:2899
6672023-12-05T05:24:10.800ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:2920
6682023-12-05T05:24:10.800ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 25331372 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:40426 req_id = da168ef0-44f5-4142-b4bb-e028cfac65a0 response_code = 204 uri = /cockroachdb
6692023-12-05T05:24:10.801ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
6702023-12-05T05:24:10.834ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd82653d-a25b-409a-af88-7344f9f8a41b, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: b96c752c-8512-47e8-bbf5-dddb77760297, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: fa05c9b8-e2a1-4c10-a933-8980828c20ab, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 7fe67b7a-7621-4dfb-a4f7-62a16105eaa7, kind: Service { id: fa05c9b8-e2a1-4c10-a933-8980828c20ab }, name: Name("ntp-fa05c9b8-e2a1-4c10-a933-8980828c20ab"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 90d6cc55-c535-4b71-a7d5-769e23015de9, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 32271630-ccab-448f-a627-a273a66ed73f, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 876fd524-b8a9-4c79-a2ea-fcf06eb6279e, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } } } }, OmicronZoneConfig { id: a48280e1-9a76-43b1-b5d2-6ba2da05e4d5, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } } } }, OmicronZoneConfig { id: fa06112d-8305-4f51-bbf3-54d846151c2a, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } } } }, OmicronZoneConfig { id: b48f303f-75ee-4094-966f-886d1e478875, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 4c94cd73-01d8-4739-9581-87c53de995f7, kind: Service { id: b48f303f-75ee-4094-966f-886d1e478875 }, name: Name("external-dns-b48f303f-75ee-4094-966f-886d1e478875"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 113])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: e08d2920-4c89-42d9-9053-73f77c061a0b, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: cd0ae421-66bb-4be3-b909-2b2ef27442ce, kind: Service { id: e08d2920-4c89-42d9-9053-73f77c061a0b }, name: Name("external-dns-e08d2920-4c89-42d9-9053-73f77c061a0b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 7])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6211c8ed-e16d-4690-bbc9-c6b0b0e40bce, kind: Service { id: 12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1 }, name: Name("nexus-12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1563d93c-b3b5-40ac-8054-4a91e28c9460, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 36a8119d-ea90-4878-952b-0e609d0b4a49, kind: Service { id: 1563d93c-b3b5-40ac-8054-4a91e28c9460 }, name: Name("nexus-1563d93c-b3b5-40ac-8054-4a91e28c9460"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 6])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4e244751-b66e-475b-b63b-0d85745ef52e, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 6ba54a5c-dca3-4767-926d-d030768bef08, kind: Service { id: 4e244751-b66e-475b-b63b-0d85745ef52e }, name: Name("nexus-4e244751-b66e-475b-b63b-0d85745ef52e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 251, 107])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: ba5f27cd-194b-4f05-bbe9-232972a7f9d6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 175bf0a4-30e0-41d1-a8c7-4f0566d06222, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: d7b164bd-8476-4001-b9c9-a3c2e6ac4aff, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: d65073c7-ffd3-4302-b947-4b4309b78545, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 0c30cdc2-a536-455b-abad-80009a7fcb5b, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 7a31351c-813d-4593-a8fb-3bfaf5fb620b, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: afdf521b-66b8-466f-aa2e-7f75ad3e2e99, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 9a248a0b-25c8-4828-916d-6bd74a5d85fe, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External } } } }, OmicronZoneConfig { id: ed93e1af-3b69-475e-a2d3-db5b6e80c3ed, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External } } } }, OmicronZoneConfig { id: 604be57f-463f-4189-a216-aa7e1028a594, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External } } } }, OmicronZoneConfig { id: f15c28fb-6291-412b-a10e-fe9d57ee2a69, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External } } } }, OmicronZoneConfig { id: 6cf8cdef-26af-4693-b210-bbd682890a02, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External } } } }, OmicronZoneConfig { id: 6877d4e8-0a5a-4668-9655-af17ac189fce, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }, OmicronZoneConfig { id: 4f1777b4-0257-4333-9bc4-04112a7947c4, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:299 sled_address = [fd00:1122:3344:101::1]:12345
6712023-12-05T05:24:10.835ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53543
6722023-12-05T05:24:10.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6732023-12-05T05:24:10.835ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: bd82653d-a25b-409a-af88-7344f9f8a41b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6742023-12-05T05:24:10.858ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6752023-12-05T05:24:10.858ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: b96c752c-8512-47e8-bbf5-dddb77760297, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6762023-12-05T05:24:10.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6772023-12-05T05:24:10.879ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 44ac5ff3-cccf-4698-a3c5-2fe40573f2cc, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: InternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6782023-12-05T05:24:10.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 90d6cc55-c535-4b71-a7d5-769e23015de9, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6792023-12-05T05:24:10.897ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 90d6cc55-c535-4b71-a7d5-769e23015de9, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6802023-12-05T05:24:10.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32271630-ccab-448f-a627-a273a66ed73f, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6812023-12-05T05:24:10.916ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 32271630-ccab-448f-a627-a273a66ed73f, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6822023-12-05T05:24:10.939ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 876fd524-b8a9-4c79-a2ea-fcf06eb6279e, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6832023-12-05T05:24:10.939ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 876fd524-b8a9-4c79-a2ea-fcf06eb6279e, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6842023-12-05T05:24:10.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a48280e1-9a76-43b1-b5d2-6ba2da05e4d5, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6852023-12-05T05:24:10.961ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: a48280e1-9a76-43b1-b5d2-6ba2da05e4d5, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6862023-12-05T05:24:10.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fa06112d-8305-4f51-bbf3-54d846151c2a, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6872023-12-05T05:24:10.986ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: fa06112d-8305-4f51-bbf3-54d846151c2a, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: CockroachDb }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6882023-12-05T05:24:11.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b48f303f-75ee-4094-966f-886d1e478875, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6892023-12-05T05:24:11.005ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: b48f303f-75ee-4094-966f-886d1e478875, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6902023-12-05T05:24:11.152ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e08d2920-4c89-42d9-9053-73f77c061a0b, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6912023-12-05T05:24:11.155ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: e08d2920-4c89-42d9-9053-73f77c061a0b, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: ExternalDns }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6922023-12-05T05:24:11.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 175bf0a4-30e0-41d1-a8c7-4f0566d06222, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Clickhouse }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6932023-12-05T05:24:11.265ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 175bf0a4-30e0-41d1-a8c7-4f0566d06222, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Clickhouse }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6942023-12-05T05:24:11.397ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7a31351c-813d-4593-a8fb-3bfaf5fb620b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6952023-12-05T05:24:11.398ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 7a31351c-813d-4593-a8fb-3bfaf5fb620b, dataset_name: DatasetName { pool_name: ZpoolName { id: 14b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6962023-12-05T05:24:11.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: afdf521b-66b8-466f-aa2e-7f75ad3e2e99, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6972023-12-05T05:24:11.482ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: afdf521b-66b8-466f-aa2e-7f75ad3e2e99, dataset_name: DatasetName { pool_name: ZpoolName { id: 24b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
6982023-12-05T05:24:11.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a248a0b-25c8-4828-916d-6bd74a5d85fe, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
6992023-12-05T05:24:11.594ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 9a248a0b-25c8-4828-916d-6bd74a5d85fe, dataset_name: DatasetName { pool_name: ZpoolName { id: 31bd71cd-4736-4a12-a387-9b74b050396f, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7002023-12-05T05:24:11.643ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61269
7012023-12-05T05:24:11.643ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 108 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61269 req_id = 5d8a6903-75bc-4e0c-a038-1fb5fe2b1987 response_code = 200 uri = /baseboard
7022023-12-05T05:24:11.723ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ed93e1af-3b69-475e-a2d3-db5b6e80c3ed, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7032023-12-05T05:24:11.723ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: ed93e1af-3b69-475e-a2d3-db5b6e80c3ed, dataset_name: DatasetName { pool_name: ZpoolName { id: 616b26df-e62a-4c68-b506-f4a923d8aaf7, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7042023-12-05T05:24:11.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 604be57f-463f-4189-a216-aa7e1028a594, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7052023-12-05T05:24:11.816ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 604be57f-463f-4189-a216-aa7e1028a594, dataset_name: DatasetName { pool_name: ZpoolName { id: cd70d7f6-2354-4bf2-8012-55bf9eaf7930, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7062023-12-05T05:24:11.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f15c28fb-6291-412b-a10e-fe9d57ee2a69, dataset_name: DatasetName { pool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7072023-12-05T05:24:11.901ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: f15c28fb-6291-412b-a10e-fe9d57ee2a69, dataset_name: DatasetName { pool_name: ZpoolName { id: ceb4461c-cf56-4719-ad3c-14430bfdfb60, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7082023-12-05T05:24:12.020ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6cf8cdef-26af-4693-b210-bbd682890a02, dataset_name: DatasetName { pool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7092023-12-05T05:24:12.022ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 6cf8cdef-26af-4693-b210-bbd682890a02, dataset_name: DatasetName { pool_name: ZpoolName { id: d462a7f7-b628-40fe-80ff-4e4189e2d62b, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7102023-12-05T05:24:12.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6877d4e8-0a5a-4668-9655-af17ac189fce, dataset_name: DatasetName { pool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7112023-12-05T05:24:12.111ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 6877d4e8-0a5a-4668-9655-af17ac189fce, dataset_name: DatasetName { pool_name: ZpoolName { id: e4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7122023-12-05T05:24:12.183ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f1777b4-0257-4333-9bc4-04112a7947c4, dataset_name: DatasetName { pool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } }) file = sled-storage/src/manager.rs:346
7132023-12-05T05:24:12.183ZINFOSledAgent (StorageManager): add_dataset: NewFilesystemRequest { dataset_id: 4f1777b4-0257-4333-9bc4-04112a7947c4, dataset_name: DatasetName { pool_name: ZpoolName { id: f4b4dc87-ab46-49fb-a4b4-d361ae214c03, kind: External }, kind: Crucible }, responder: Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) } } file = sled-storage/src/manager.rs:564
7142023-12-05T05:24:12.271ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7152023-12-05T05:24:12.271ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
7162023-12-05T05:24:12.271ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/omicron-zones.json file = common/src/ledger.rs:176
7172023-12-05T05:24:12.271ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7182023-12-05T05:24:12.275ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7192023-12-05T05:24:12.275ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7202023-12-05T05:24:12.277ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7212023-12-05T05:24:12.284ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7222023-12-05T05:24:12.284ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7232023-12-05T05:24:12.284ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7242023-12-05T05:24:12.284ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7252023-12-05T05:24:12.284ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7262023-12-05T05:24:12.284ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7272023-12-05T05:24:12.288ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7282023-12-05T05:24:12.288ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7292023-12-05T05:24:12.288ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7302023-12-05T05:24:12.291ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:253
7312023-12-05T05:24:12.291ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7322023-12-05T05:24:12.297ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7332023-12-05T05:24:12.297ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7342023-12-05T05:24:12.297ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7352023-12-05T05:24:12.297ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7362023-12-05T05:24:12.297ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7372023-12-05T05:24:12.567ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4 file = illumos-utils/src/zone.rs:314
7382023-12-05T05:24:12.572ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222 file = illumos-utils/src/zone.rs:314
7392023-12-05T05:24:12.576ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b file = illumos-utils/src/zone.rs:314
7402023-12-05T05:24:12.581ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594 file = illumos-utils/src/zone.rs:314
7412023-12-05T05:24:12.586ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff file = illumos-utils/src/zone.rs:314
7422023-12-05T05:24:12.590ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02 file = illumos-utils/src/zone.rs:314
7432023-12-05T05:24:12.596ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69 file = illumos-utils/src/zone.rs:314
7442023-12-05T05:24:12.600ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce file = illumos-utils/src/zone.rs:314
7452023-12-05T05:24:12.605ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6 file = illumos-utils/src/zone.rs:314
7462023-12-05T05:24:12.610ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4 file = illumos-utils/src/zone.rs:353
7472023-12-05T05:24:12.615ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222 file = illumos-utils/src/zone.rs:353
7482023-12-05T05:24:12.620ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b file = illumos-utils/src/zone.rs:353
7492023-12-05T05:24:12.625ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594 file = illumos-utils/src/zone.rs:353
7502023-12-05T05:24:12.629ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff file = illumos-utils/src/zone.rs:353
7512023-12-05T05:24:12.633ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02 file = illumos-utils/src/zone.rs:353
7522023-12-05T05:24:12.638ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69 file = illumos-utils/src/zone.rs:353
7532023-12-05T05:24:12.644ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce file = illumos-utils/src/zone.rs:353
7542023-12-05T05:24:12.649ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6 file = illumos-utils/src/zone.rs:353
7552023-12-05T05:24:12.655ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545 file = illumos-utils/src/zone.rs:314
7562023-12-05T05:24:12.661ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe file = illumos-utils/src/zone.rs:314
7572023-12-05T05:24:12.666ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b file = illumos-utils/src/zone.rs:314
7582023-12-05T05:24:12.672ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed file = illumos-utils/src/zone.rs:314
7592023-12-05T05:24:12.679ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99 file = illumos-utils/src/zone.rs:314
7602023-12-05T05:24:12.685ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545 file = illumos-utils/src/zone.rs:353
7612023-12-05T05:24:12.691ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe file = illumos-utils/src/zone.rs:353
7622023-12-05T05:24:12.697ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b file = illumos-utils/src/zone.rs:353
7632023-12-05T05:24:12.702ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed file = illumos-utils/src/zone.rs:353
7642023-12-05T05:24:12.707ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99 file = illumos-utils/src/zone.rs:353
7652023-12-05T05:24:12.750ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7662023-12-05T05:24:12.784ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7672023-12-05T05:24:12.826ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7682023-12-05T05:24:12.862ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7692023-12-05T05:24:12.897ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:270
7702023-12-05T05:24:12.899ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7712023-12-05T05:24:12.899ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7722023-12-05T05:24:12.954ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte1", _ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 161, 7]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte1" } }
7732023-12-05T05:24:12.954ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = external_dns
7742023-12-05T05:24:12.958ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7752023-12-05T05:24:12.958ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7762023-12-05T05:24:13.009ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte2", _ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 251, 107]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte2" } }
7772023-12-05T05:24:13.009ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = nexus
7782023-12-05T05:24:13.011ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7792023-12-05T05:24:13.011ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7802023-12-05T05:24:13.059ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte3", _ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 128, 6]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte3" } }
7812023-12-05T05:24:13.059ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = nexus
7822023-12-05T05:24:13.061ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7832023-12-05T05:24:13.061ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7842023-12-05T05:24:13.108ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte4", _ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 228, 1]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte4" } }
7852023-12-05T05:24:13.108ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = nexus
7862023-12-05T05:24:13.110ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:295
7872023-12-05T05:24:13.110ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:187
7882023-12-05T05:24:13.162ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte5", _ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 206, 113]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte5" } }
7892023-12-05T05:24:13.162ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1226 zone_type = external_dns
7902023-12-05T05:24:13.163ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
7912023-12-05T05:24:13.163ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
7922023-12-05T05:24:13.164ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
7932023-12-05T05:24:13.168ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
7942023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
7952023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
7962023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
7972023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
7982023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
7992023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
8002023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:98
8012023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:105
8022023-12-05T05:24:13.194ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8032023-12-05T05:24:13.198ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8042023-12-05T05:24:13.200ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8052023-12-05T05:24:13.200ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8062023-12-05T05:24:13.200ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8072023-12-05T05:24:13.202ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:133
8082023-12-05T05:24:13.204ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8092023-12-05T05:24:13.204ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
8102023-12-05T05:24:13.289ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b file = illumos-utils/src/zone.rs:314
8112023-12-05T05:24:13.294ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e file = illumos-utils/src/zone.rs:314
8122023-12-05T05:24:13.298ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460 file = illumos-utils/src/zone.rs:314
8132023-12-05T05:24:13.304ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1 file = illumos-utils/src/zone.rs:314
8142023-12-05T05:24:13.310ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875 file = illumos-utils/src/zone.rs:314
8152023-12-05T05:24:13.315ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b file = illumos-utils/src/zone.rs:353
8162023-12-05T05:24:13.322ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e file = illumos-utils/src/zone.rs:353
8172023-12-05T05:24:13.328ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460 file = illumos-utils/src/zone.rs:353
8182023-12-05T05:24:13.333ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1 file = illumos-utils/src/zone.rs:353
8192023-12-05T05:24:13.339ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875 file = illumos-utils/src/zone.rs:353
8202023-12-05T05:24:15.804ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
8212023-12-05T05:24:16.556ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="9bed7a56-bf67-47f3-b217-179cf2275832"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8222023-12-05T05:24:16.556ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8232023-12-05T05:24:16.669ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="fac5c2bf-5f61-4dca-be95-5e8421b0f77d"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8242023-12-05T05:24:16.679ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8252023-12-05T05:24:16.747ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8262023-12-05T05:24:16.879ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="bded68b6-f728-45a0-9300-c17c2dda472a"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8272023-12-05T05:24:16.880ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8282023-12-05T05:24:17.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 63.430741ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8292023-12-05T05:24:17.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 49.940598ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8302023-12-05T05:24:17.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 46.97536ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8312023-12-05T05:24:17.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 58.232502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8322023-12-05T05:24:17.455ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
8332023-12-05T05:24:17.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 140.095056ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8342023-12-05T05:24:17.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 57.666028ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8352023-12-05T05:24:17.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 108.39121ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8362023-12-05T05:24:17.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 145.993974ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8372023-12-05T05:24:17.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 290.99343ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8382023-12-05T05:24:17.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 175.877814ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8392023-12-05T05:24:17.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 130.593865ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8402023-12-05T05:24:17.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 240.122421ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8412023-12-05T05:24:17.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 61.686202ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
8422023-12-05T05:24:17.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 568.224917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8432023-12-05T05:24:17.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 65.374201ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
8442023-12-05T05:24:17.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 292.973621ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8452023-12-05T05:24:17.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 441.058447ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8462023-12-05T05:24:17.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 196.075287ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
8472023-12-05T05:24:17.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 498.388494ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8482023-12-05T05:24:18.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 388.70527ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
8492023-12-05T05:24:18.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 879.744836ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8502023-12-05T05:24:18.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 723.198897ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8512023-12-05T05:24:18.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 1.140896163s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8522023-12-05T05:24:18.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 536.869738ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8532023-12-05T05:24:18.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 821.107438ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
8542023-12-05T05:24:19.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 502.526806ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8552023-12-05T05:24:19.077ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 941.116725ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8562023-12-05T05:24:19.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 1.408892566s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8572023-12-05T05:24:19.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 1.467496504s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
8582023-12-05T05:24:19.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 814.958427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8592023-12-05T05:24:19.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 684.440719ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8602023-12-05T05:24:19.814ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
8612023-12-05T05:24:19.871ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8622023-12-05T05:24:19.872ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
8632023-12-05T05:24:19.880ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8642023-12-05T05:24:19.880ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
8652023-12-05T05:24:19.910ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="ab7cacfa-40cb-4bfa-9de8-7e8a6df77f34"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8662023-12-05T05:24:19.910ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
8672023-12-05T05:24:19.934ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="878a9620-8ac6-44ff-8288-664603f24b1c"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8682023-12-05T05:24:19.934ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
8692023-12-05T05:24:20.017ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="c978181f-a338-43b5-af43-169d68b6a90d"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8702023-12-05T05:24:20.017ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
8712023-12-05T05:24:20.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 1.265245425s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
8722023-12-05T05:24:20.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 51.631933ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
8732023-12-05T05:24:20.054ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="86d8c4fe-dc64-46a8-99f6-c9ab0f643415"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8742023-12-05T05:24:20.054ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
8752023-12-05T05:24:20.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 70.479629ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
8762023-12-05T05:24:20.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 59.452363ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
8772023-12-05T05:24:20.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 95.787426ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
8782023-12-05T05:24:20.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 59.723572ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
8792023-12-05T05:24:20.170ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8802023-12-05T05:24:20.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 69.984692ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
8812023-12-05T05:24:20.248ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
8822023-12-05T05:24:20.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 30.92153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
8832023-12-05T05:24:20.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 112.485738ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
8842023-12-05T05:24:20.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 121.425815ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
8852023-12-05T05:24:20.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 520.977292ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
8862023-12-05T05:24:20.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 125.808574ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
8872023-12-05T05:24:20.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 38.043437ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
8882023-12-05T05:24:20.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 117.850558ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
8892023-12-05T05:24:20.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 695.746023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
8902023-12-05T05:24:20.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 59.117889ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
8912023-12-05T05:24:20.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 125.01875ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
8922023-12-05T05:24:20.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 369.817591ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
8932023-12-05T05:24:20.809ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
8942023-12-05T05:24:20.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 755.73752ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
8952023-12-05T05:24:20.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 254.93649ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
8962023-12-05T05:24:20.883ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_31bd71cd-4736-4a12-a387-9b74b050396f/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="8a9c2416-f9cd-4b35-ae14-492dd42de0a1"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8972023-12-05T05:24:20.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 70.116531ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
8982023-12-05T05:24:20.969ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="dataset" value="oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="7850c53f-ffd8-4233-8410-056e2cb30bc0"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8992023-12-05T05:24:20.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 147.812988ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
9002023-12-05T05:24:20.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 109.243817ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
9012023-12-05T05:24:21.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 116.27801ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
9022023-12-05T05:24:21.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 65.680669ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
9032023-12-05T05:24:21.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 239.013067ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
9042023-12-05T05:24:21.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 1.268934425s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
9052023-12-05T05:24:21.043ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9062023-12-05T05:24:21.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 78.409392ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
9072023-12-05T05:24:21.057ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
9082023-12-05T05:24:21.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 282.338157ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
9092023-12-05T05:24:21.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 282.525855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
9102023-12-05T05:24:21.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 552.955638ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
9112023-12-05T05:24:21.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 550.505527ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
9122023-12-05T05:24:21.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 1.011023736s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
9132023-12-05T05:24:21.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 958.856801ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
9142023-12-05T05:24:21.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 238.186182ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
9152023-12-05T05:24:21.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 1.119538561s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
9162023-12-05T05:24:21.217ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9172023-12-05T05:24:21.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 235.890938ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
9182023-12-05T05:24:21.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 62.004321ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9192023-12-05T05:24:21.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 53.527622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
9202023-12-05T05:24:21.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 1.180658775s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
9212023-12-05T05:24:21.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 108.755043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9222023-12-05T05:24:21.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 79.578099ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
9232023-12-05T05:24:21.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 247.145216ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
9242023-12-05T05:24:21.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 1.310227176s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
9252023-12-05T05:24:21.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 577.328826ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
9262023-12-05T05:24:21.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 331.928709ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
9272023-12-05T05:24:21.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 205.966933ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
9282023-12-05T05:24:21.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 265.785416ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9292023-12-05T05:24:21.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 25.184856ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9302023-12-05T05:24:21.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 116.238562ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9312023-12-05T05:24:21.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 1.063125587s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
9322023-12-05T05:24:21.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 637.111096ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
9332023-12-05T05:24:21.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 738.449525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
9342023-12-05T05:24:21.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 122.193422ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9352023-12-05T05:24:21.752ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
9362023-12-05T05:24:21.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 793.825521ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
9372023-12-05T05:24:21.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 909.663467ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
9382023-12-05T05:24:21.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 282.692772ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
9392023-12-05T05:24:21.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 422.788619ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9402023-12-05T05:24:21.951ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
9412023-12-05T05:24:21.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 282.922004ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9422023-12-05T05:24:21.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 998.185836ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
9432023-12-05T05:24:22.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 593.442658ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
9442023-12-05T05:24:22.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 28.500465ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
9452023-12-05T05:24:22.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 66.711801ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
9462023-12-05T05:24:22.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 1.297495342s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
9472023-12-05T05:24:22.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 666.275076ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
9482023-12-05T05:24:22.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 45.904284ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
9492023-12-05T05:24:22.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 616.560273ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
9502023-12-05T05:24:22.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 674.125454ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9512023-12-05T05:24:22.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 829.061827ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
9522023-12-05T05:24:22.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 1.466230815s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
9532023-12-05T05:24:22.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 116.352178ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
9542023-12-05T05:24:22.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 55.338824ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
9552023-12-05T05:24:22.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 649.39154ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
9562023-12-05T05:24:22.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 583.982558ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9572023-12-05T05:24:22.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 200.617263ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
9582023-12-05T05:24:22.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 1.439294767s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
9592023-12-05T05:24:22.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 246.423544ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
9602023-12-05T05:24:22.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 1.189624655s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
9612023-12-05T05:24:22.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 523.558113ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
9622023-12-05T05:24:22.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 1.454596804s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
9632023-12-05T05:24:22.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 1.485592779s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
9642023-12-05T05:24:22.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 377.663019ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
9652023-12-05T05:24:22.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 1.035744703s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
9662023-12-05T05:24:22.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 880.671474ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
9672023-12-05T05:24:22.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 1.216097708s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
9682023-12-05T05:24:22.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 1.205428173s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
9692023-12-05T05:24:22.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 596.373555ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
9702023-12-05T05:24:22.937ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9712023-12-05T05:24:22.937ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
9722023-12-05T05:24:23.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 648.429043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9732023-12-05T05:24:23.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 553.661849ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9742023-12-05T05:24:23.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 701.160552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
9752023-12-05T05:24:23.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 944.931173ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
9762023-12-05T05:24:23.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 837.043038ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
9772023-12-05T05:24:23.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 982.878357ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
9782023-12-05T05:24:23.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 1.153692983s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
9792023-12-05T05:24:23.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 38.416609ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
9802023-12-05T05:24:23.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 113.501656ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
9812023-12-05T05:24:23.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 1.072692809s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
9822023-12-05T05:24:23.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 150.441332ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
9832023-12-05T05:24:23.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 1.162183643s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
9842023-12-05T05:24:23.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 644.41062ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
9852023-12-05T05:24:23.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 569.985489ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
9862023-12-05T05:24:23.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 965.613234ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
9872023-12-05T05:24:23.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 975.717089ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
9882023-12-05T05:24:23.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 1.423815302s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
9892023-12-05T05:24:23.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 933.28217ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
9902023-12-05T05:24:23.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 806.910955ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
9912023-12-05T05:24:23.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 851.405328ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
9922023-12-05T05:24:23.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 622.755897ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
9932023-12-05T05:24:23.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 932.424739ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
9942023-12-05T05:24:24.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 1.302433273s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
9952023-12-05T05:24:24.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 1.190006249s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
9962023-12-05T05:24:24.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 1.050321716s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
9972023-12-05T05:24:24.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 502.01666ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
9982023-12-05T05:24:24.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 1.17157556s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
9992023-12-05T05:24:24.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 1.38400977s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
10002023-12-05T05:24:24.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 738.74728ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
10012023-12-05T05:24:24.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 657.441049ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
10022023-12-05T05:24:24.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 657.048965ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
10032023-12-05T05:24:24.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 1.194368976s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
10042023-12-05T05:24:24.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 1.030363456s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
10052023-12-05T05:24:24.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 827.027067ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
10062023-12-05T05:24:24.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 557.813981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
10072023-12-05T05:24:24.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 643.608178ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
10082023-12-05T05:24:24.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 584.67977ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
10092023-12-05T05:24:24.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 509.991394ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
10102023-12-05T05:24:24.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 1.168467816s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
10112023-12-05T05:24:24.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 1.253636255s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
10122023-12-05T05:24:24.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 1.272380854s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
10132023-12-05T05:24:24.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 678.185392ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
10142023-12-05T05:24:24.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 671.127725ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
10152023-12-05T05:24:25.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 920.972709ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
10162023-12-05T05:24:25.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 820.339011ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
10172023-12-05T05:24:25.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 1.083962482s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
10182023-12-05T05:24:25.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 632.242667ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
10192023-12-05T05:24:25.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 934.002949ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
10202023-12-05T05:24:25.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 1.343821034s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
10212023-12-05T05:24:25.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 1.266662046s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
10222023-12-05T05:24:25.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 651.335213ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
10232023-12-05T05:24:25.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 1.39007997s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
10242023-12-05T05:24:25.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 1.105207979s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
10252023-12-05T05:24:25.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 1.494163448s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
10262023-12-05T05:24:25.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 876.211168ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
10272023-12-05T05:24:25.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 768.398048ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
10282023-12-05T05:24:25.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 1.283658887s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
10292023-12-05T05:24:25.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 681.096917ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
10302023-12-05T05:24:25.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 1.271654392s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
10312023-12-05T05:24:25.811ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
10322023-12-05T05:24:25.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 1.489902377s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
10332023-12-05T05:24:25.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 639.027107ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
10342023-12-05T05:24:25.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 736.303863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
10352023-12-05T05:24:25.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 898.300752ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
10362023-12-05T05:24:26.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 931.968279ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
10372023-12-05T05:24:26.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 942.401426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
10382023-12-05T05:24:26.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 1.398979472s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
10392023-12-05T05:24:26.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 1.244384898s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
10402023-12-05T05:24:26.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 1.086656303s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
10412023-12-05T05:24:26.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 608.137795ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
10422023-12-05T05:24:26.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 1.418898458s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
10432023-12-05T05:24:26.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 1.457393453s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
10442023-12-05T05:24:26.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 625.514136ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
10452023-12-05T05:24:26.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875") failed: Property not found. retry in 1.452865471s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
10462023-12-05T05:24:26.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 692.078153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
10472023-12-05T05:24:26.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff") failed: Property not found. retry in 1.074208217s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d7b164bd-8476-4001-b9c9-a3c2e6ac4aff
10482023-12-05T05:24:26.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 1.048667582s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
10492023-12-05T05:24:26.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 922.131383ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
10502023-12-05T05:24:26.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 920.131031ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
10512023-12-05T05:24:26.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b") failed: Property not found. retry in 1.403618988s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
10522023-12-05T05:24:26.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 1.056162629s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
10532023-12-05T05:24:26.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b") failed: Property not found. retry in 1.438851474s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_0c30cdc2-a536-455b-abad-80009a7fcb5b
10542023-12-05T05:24:26.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02") failed: Property not found. retry in 1.323690313s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6cf8cdef-26af-4693-b210-bbd682890a02
10552023-12-05T05:24:27.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 972.759768ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
10562023-12-05T05:24:27.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 813.656062ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
10572023-12-05T05:24:27.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1") failed: Property not found. retry in 1.156359733s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
10582023-12-05T05:24:27.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe") failed: Property not found. retry in 1.002525357s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a248a0b-25c8-4828-916d-6bd74a5d85fe
10592023-12-05T05:24:27.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 679.00643ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
10602023-12-05T05:24:27.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4") failed: Property not found. retry in 1.324518257s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_4f1777b4-0257-4333-9bc4-04112a7947c4
10612023-12-05T05:24:27.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460") failed: Property not found. retry in 936.750697ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
10622023-12-05T05:24:27.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99") failed: Property not found. retry in 1.335836786s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_afdf521b-66b8-466f-aa2e-7f75ad3e2e99
10632023-12-05T05:24:27.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b") failed: Property not found. retry in 1.393948353s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7a31351c-813d-4593-a8fb-3bfaf5fb620b
10642023-12-05T05:24:27.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed") failed: Property not found. retry in 872.519785ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed93e1af-3b69-475e-a2d3-db5b6e80c3ed
10652023-12-05T05:24:27.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e") failed: Property not found. retry in 1.046948504s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
10662023-12-05T05:24:27.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69") failed: Property not found. retry in 565.400663ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f15c28fb-6291-412b-a10e-fe9d57ee2a69
10672023-12-05T05:24:27.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545") failed: Property not found. retry in 1.357816795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d65073c7-ffd3-4302-b947-4b4309b78545
10682023-12-05T05:24:27.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594") failed: Property not found. retry in 554.770656ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_604be57f-463f-4189-a216-aa7e1028a594
10692023-12-05T05:24:28.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222") failed: Property not found. retry in 1.258389611s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_175bf0a4-30e0-41d1-a8c7-4f0566d06222
10702023-12-05T05:24:28.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6") failed: Property not found. retry in 1.356424936s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
10712023-12-05T05:24:28.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce") failed: Property not found. retry in 967.061087ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6877d4e8-0a5a-4668-9655-af17ac189fce
10722023-12-05T05:24:28.144ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::8 exists file = sled-agent/src/services.rs:1769
10732023-12-05T05:24:28.144ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::8, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
10742023-12-05T05:24:28.801ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::8 exists - OK file = sled-agent/src/services.rs:1777
10752023-12-05T05:24:28.801ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
10762023-12-05T05:24:29.630ZINFOSledAgent (ServiceManager): Setting up external-dns service file = sled-agent/src/services.rs:1953
10772023-12-05T05:24:29.630ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_external_dns_b48f303f-75ee-4094-966f-886d1e478875
10782023-12-05T05:24:30.812ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
10792023-12-05T05:24:34.744ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::9 exists file = sled-agent/src/services.rs:1769
10802023-12-05T05:24:34.744ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::9, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
10812023-12-05T05:24:35.296ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::9 exists - OK file = sled-agent/src/services.rs:1777
10822023-12-05T05:24:35.296ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
10832023-12-05T05:24:35.486ZINFOSledAgent (ServiceManager): Setting up external-dns service file = sled-agent/src/services.rs:1953
10842023-12-05T05:24:35.486ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_external_dns_e08d2920-4c89-42d9-9053-73f77c061a0b
10852023-12-05T05:24:35.813ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
10862023-12-05T05:24:40.814ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
10872023-12-05T05:24:41.063ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::c exists file = sled-agent/src/services.rs:1769
10882023-12-05T05:24:41.063ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::c, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
10892023-12-05T05:24:41.621ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::c exists - OK file = sled-agent/src/services.rs:1777
10902023-12-05T05:24:41.622ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
10912023-12-05T05:24:41.674ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33899
10922023-12-05T05:24:41.674ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 90 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33899 req_id = 4ed49291-b40e-487b-8325-dbfd25c6f7bc response_code = 200 uri = /baseboard
10932023-12-05T05:24:41.823ZINFOSledAgent (ServiceManager): Setting up Nexus service file = sled-agent/src/services.rs:1840
10942023-12-05T05:24:41.823ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_nexus_4e244751-b66e-475b-b63b-0d85745ef52e
10952023-12-05T05:24:45.814ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
10962023-12-05T05:24:47.619ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::b exists file = sled-agent/src/services.rs:1769
10972023-12-05T05:24:47.619ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::b, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
10982023-12-05T05:24:48.178ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::b exists - OK file = sled-agent/src/services.rs:1777
10992023-12-05T05:24:48.180ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11002023-12-05T05:24:48.386ZINFOSledAgent (ServiceManager): Setting up Nexus service file = sled-agent/src/services.rs:1840
11012023-12-05T05:24:48.386ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_nexus_1563d93c-b3b5-40ac-8054-4a91e28c9460
11022023-12-05T05:24:50.815ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11032023-12-05T05:24:53.414ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::a exists file = sled-agent/src/services.rs:1769
11042023-12-05T05:24:53.414ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::a, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
11052023-12-05T05:24:53.970ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::a exists - OK file = sled-agent/src/services.rs:1777
11062023-12-05T05:24:53.970ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11072023-12-05T05:24:54.179ZINFOSledAgent (ServiceManager): Setting up Nexus service file = sled-agent/src/services.rs:1840
11082023-12-05T05:24:54.179ZINFOSledAgent (ServiceManager): Ensuring address for OPTE port file = illumos-utils/src/running_zone.rs:652 zone = oxz_nexus_12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1
11092023-12-05T05:24:55.816ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11102023-12-05T05:25:00.048ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::d exists file = sled-agent/src/services.rs:1769
11112023-12-05T05:25:00.048ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::d, prefix: 64 })) file = illumos-utils/src/running_zone.rs:608 zone = oxz_oximeter_ba5f27cd-194b-4f05-bbe9-232972a7f9d6
11122023-12-05T05:25:00.607ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::d exists - OK file = sled-agent/src/services.rs:1777
11132023-12-05T05:25:00.607ZINFOSledAgent (ServiceManager): Zone using sled underlay as gateway file = sled-agent/src/services.rs:1796
11142023-12-05T05:25:00.818ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11152023-12-05T05:25:00.826ZINFOSledAgent (ServiceManager): Setting up oximeter service file = sled-agent/src/services.rs:2064
11162023-12-05T05:25:00.914ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
11172023-12-05T05:25:00.915ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:198
11182023-12-05T05:25:00.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 50085421 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53543 req_id = 11bcdaba-0fd0-4d15-a928-d64fd0d7da23 response_code = 204 uri = /omicron-zones
11192023-12-05T05:25:00.920ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1062
11202023-12-05T05:25:00.920ZINFOSledAgent (RSS): Writing ledger to /pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:198
11212023-12-05T05:25:00.920ZINFOSledAgent (RSS): Writing ledger to /pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:198
11222023-12-05T05:25:00.924ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:549
11232023-12-05T05:25:00.924ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::a]:12221 file = sled-agent/src/rack_setup/service.rs:551
11242023-12-05T05:25:00.956ZINFOSledAgent (RSS): rack_network_config: Some( RackNetworkConfigV1 { bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV1 { addresses: [ V4( Ipv4Network { addr: 10.151.1.174, prefix: 32, }, ), ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.1.100, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Network { addr: fd00:1122:3344:1::, prefix: 56, }, }, ) file = sled-agent/src/rack_setup/service.rs:658
11252023-12-05T05:25:00.957ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/7f701f68-f023-4c8b-84b4-4cfebb9ea428/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11262023-12-05T05:25:01.096ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/7f701f68-f023-4c8b-84b4-4cfebb9ea428/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11272023-12-05T05:25:01.263ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/7f701f68-f023-4c8b-84b4-4cfebb9ea428/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11282023-12-05T05:25:01.466ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/7f701f68-f023-4c8b-84b4-4cfebb9ea428/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:679
11292023-12-05T05:25:02.873ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a10fe65c-d1ce-40f9-bd54-e49e18b4e8ad", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a10fe65c-d1ce-40f9-bd54-e49e18b4e8ad" } file = sled-agent/src/rack_setup/service.rs:679
11302023-12-05T05:25:03.143ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c6649bb4-5d8f-4fe7-b0d9-c941fad7cc90", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c6649bb4-5d8f-4fe7-b0d9-c941fad7cc90" } file = sled-agent/src/rack_setup/service.rs:679
11312023-12-05T05:25:03.444ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "83bb1336-75f2-49d3-a73e-45e08f9a7fec", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "83bb1336-75f2-49d3-a73e-45e08f9a7fec" } file = sled-agent/src/rack_setup/service.rs:679
11322023-12-05T05:25:03.743ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ccfd222b-59ae-46ca-88cc-044b0ae6a5f4", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ccfd222b-59ae-46ca-88cc-044b0ae6a5f4" } file = sled-agent/src/rack_setup/service.rs:679
11332023-12-05T05:25:04.130ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "56233dcd-5f72-4882-9f04-feb680772774", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "56233dcd-5f72-4882-9f04-feb680772774" } file = sled-agent/src/rack_setup/service.rs:679
11342023-12-05T05:25:04.545ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dde4cc6c-f109-4071-b221-2533e3544ae5", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dde4cc6c-f109-4071-b221-2533e3544ae5" } file = sled-agent/src/rack_setup/service.rs:679
11352023-12-05T05:25:05.282ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "69504823-7d4f-45de-b351-d6d9de1f1f09", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:05 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "69504823-7d4f-45de-b351-d6d9de1f1f09" } file = sled-agent/src/rack_setup/service.rs:679
11362023-12-05T05:25:05.818ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11372023-12-05T05:25:06.027ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "21d2a950-6bce-49e3-9e8f-18588ebdeff9", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:05 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "21d2a950-6bce-49e3-9e8f-18588ebdeff9" } file = sled-agent/src/rack_setup/service.rs:679
11382023-12-05T05:25:07.167ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5399b2d5-1dbb-4495-9b6e-71890008a971", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5399b2d5-1dbb-4495-9b6e-71890008a971" } file = sled-agent/src/rack_setup/service.rs:679
11392023-12-05T05:25:08.494ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dfa654cf-1d30-462a-bfcd-de597e6c3822", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dfa654cf-1d30-462a-bfcd-de597e6c3822" } file = sled-agent/src/rack_setup/service.rs:679
11402023-12-05T05:25:09.595ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ec042e4b-7e78-4f09-8357-d638d73b0a20", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ec042e4b-7e78-4f09-8357-d638d73b0a20" } file = sled-agent/src/rack_setup/service.rs:679
11412023-12-05T05:25:10.819ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11422023-12-05T05:25:11.463ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1eced333-0aa3-4cf4-b13d-d3b41784cd95", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1eced333-0aa3-4cf4-b13d-d3b41784cd95" } file = sled-agent/src/rack_setup/service.rs:679
11432023-12-05T05:25:11.708ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48272
11442023-12-05T05:25:11.709ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 39 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48272 req_id = d17e6949-7790-4417-bfc2-98728e753bc7 response_code = 200 uri = /baseboard
11452023-12-05T05:25:12.622ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "09383f92-1270-4d44-bb14-01bd8a0fab10", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "09383f92-1270-4d44-bb14-01bd8a0fab10" } file = sled-agent/src/rack_setup/service.rs:679
11462023-12-05T05:25:14.031ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e9fe45d0-ad98-48ed-8fd8-0b03595649db", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e9fe45d0-ad98-48ed-8fd8-0b03595649db" } file = sled-agent/src/rack_setup/service.rs:679
11472023-12-05T05:25:15.820ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11482023-12-05T05:25:16.388ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "25b3b03a-518d-41a1-aed2-672d7606bdaa", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "25b3b03a-518d-41a1-aed2-672d7606bdaa" } file = sled-agent/src/rack_setup/service.rs:679
11492023-12-05T05:25:19.150ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "01f4f7d6-9f06-45e5-9c22-59466c4b4855", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "01f4f7d6-9f06-45e5-9c22-59466c4b4855" } file = sled-agent/src/rack_setup/service.rs:679
11502023-12-05T05:25:20.820ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11512023-12-05T05:25:21.723ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4b9fdb94-8ba0-4235-a899-7b75a3db6814", "content-length": "124", "date": "Tue, 05 Dec 2023 05:25:21 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4b9fdb94-8ba0-4235-a899-7b75a3db6814" } file = sled-agent/src/rack_setup/service.rs:679
11522023-12-05T05:25:21.984ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"5f1a24c6-6689-4a53-aeca-b27122cd249d\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:21 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"5f1a24c6-6689-4a53-aeca-b27122cd249d\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.134298706s
11532023-12-05T05:25:21.986ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"8bbb6e8c-4895-462c-a8e5-17d9a35254f0\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:21 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"8bbb6e8c-4895-462c-a8e5-17d9a35254f0\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.136781472s
11542023-12-05T05:25:22.148ZINFOSledAgent (HardwareMonitor): contacting server nexus, registering sled baseboard = Baseboard { part_number: "i86pc", revision: 0, serial_number: "e0b9a1778185441db38d4af540b43da0" } file = sled-agent/src/sled_agent.rs:629 id = b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
11552023-12-05T05:25:22.198ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"d3c8fca7-e1e4-4361-9c9e-6417711e1012\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:22 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"d3c8fca7-e1e4-4361-9c9e-6417711e1012\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.349773006s
11562023-12-05T05:25:22.200ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"60abe035-c6a0-429b-9136-0f0e4aa745c3\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:21 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"60abe035-c6a0-429b-9136-0f0e4aa745c3\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.351123716s
11572023-12-05T05:25:22.298ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"2531f3c8-18e2-4294-ad69-f4281dbd5338\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:21 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"2531f3c8-18e2-4294-ad69-f4281dbd5338\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.4491739s
11582023-12-05T05:25:22.320ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"8f61d9ef-87d4-4335-a801-bfb20d264d90\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:21 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"8f61d9ef-87d4-4335-a801-bfb20d264d90\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.471516871s
11592023-12-05T05:25:22.322ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"effb26e9-bb46-40e2-8036-3573943e4723\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:21 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"effb26e9-bb46-40e2-8036-3573943e4723\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.472352889s
11602023-12-05T05:25:22.323ZWARNSledAgent (StorageMonitor): failed to notify nexus about a new pool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b on disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", model: "synthetic-model" } err = "Error Response: status: 404 Not Found; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"04e2af28-ec8d-4f07-b6a5-e56dddd218a4\\", \\"content-length\\": \\"272\\", \\"date\\": \\"Tue, 05 Dec 2023 05:25:21 GMT\\"}; value: Error { error_code: Some(\\"ObjectNotFound\\"), message: \\"not found: physical-disk with id \\\\\\"vendor = \\\\\\"synthetic-vendor\\\\\\", serial = \\\\\\"synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b\\\\\\", model = \\\\\\"synthetic-model\\\\\\"\\\\\\"\\", request_id: \\"04e2af28-ec8d-4f07-b6a5-e56dddd218a4\\" }" file = sled-agent/src/storage_monitor.rs:273 total duration = 156.474346075s
11612023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11622023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11632023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11642023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11652023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: M2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11662023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: M2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11672023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11682023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11692023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11702023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11712023-12-05T05:25:22.475ZINFOSledAgent (StorageMonitor): Nexus notification complete: Put(PhysicalDiskPutRequest { model: "synthetic-model", serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", sled_id: b21dc246-5dfb-4089-9a4d-5d9e5bd4a384, variant: U2, vendor: "synthetic-vendor" }) file = sled-agent/src/storage_monitor.rs:105
11722023-12-05T05:25:25.809ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-14b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11732023-12-05T05:25:25.821ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11742023-12-05T05:25:26.249ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "0f35cf74-92d8-409a-8b56-1ab371ba333f", "content-length": "174", "date": "Tue, 05 Dec 2023 05:25:25 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: zpool with id \"24b4dc87-ab46-49fb-a4b4-d361ae214c03\"", request_id: "0f35cf74-92d8-409a-8b56-1ab371ba333f" } file = sled-agent/src/rack_setup/service.rs:679
11752023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-24b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11762023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-31bd71cd-4736-4a12-a387-9b74b050396f", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11772023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-616b26df-e62a-4c68-b506-f4a923d8aaf7", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11782023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-a462a7f7-b628-40fe-80ff-4e4189e2d62b", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11792023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-b462a7f7-b628-40fe-80ff-4e4189e2d62b", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11802023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-cd70d7f6-2354-4bf2-8012-55bf9eaf7930", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11812023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-ceb4461c-cf56-4719-ad3c-14430bfdfb60", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11822023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-d462a7f7-b628-40fe-80ff-4e4189e2d62b", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11832023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11842023-12-05T05:25:27.049ZINFOSledAgent (StorageMonitor): Nexus notification complete: ZpoolPutRequest { disk_model: "synthetic-model", disk_serial: "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", disk_vendor: "synthetic-vendor", size: ByteCount(20937965568) } file = sled-agent/src/storage_monitor.rs:105
11852023-12-05T05:25:30.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33825
11862023-12-05T05:25:30.768ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:399 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 4c94cd73-01d8-4739-9581-87c53de995f7, kind: Service { id: b48f303f-75ee-4094-966f-886d1e478875 }, name: Name("external-dns-b48f303f-75ee-4094-966f-886d1e478875"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 113])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 1 }, NetworkInterface { id: cd0ae421-66bb-4be3-b909-2b2ef27442ce, kind: Service { id: e08d2920-4c89-42d9-9053-73f77c061a0b }, name: Name("external-dns-e08d2920-4c89-42d9-9053-73f77c061a0b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 7])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 1 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 36a8119d-ea90-4878-952b-0e609d0b4a49, kind: Service { id: 1563d93c-b3b5-40ac-8054-4a91e28c9460 }, name: Name("nexus-1563d93c-b3b5-40ac-8054-4a91e28c9460"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 6])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 1 }, NetworkInterface { id: 6211c8ed-e16d-4690-bbc9-c6b0b0e40bce, kind: Service { id: 12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1 }, name: Name("nexus-12030b1f-d84d-44ed-b4e3-0bcc9ebcb5f1"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 228, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 1 }, NetworkInterface { id: 6ba54a5c-dca3-4767-926d-d030768bef08, kind: Service { id: 4e244751-b66e-475b-b63b-0d85745ef52e }, name: Name("nexus-4e244751-b66e-475b-b63b-0d85745ef52e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 251, 107])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 1 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
11872023-12-05T05:25:30.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
11882023-12-05T05:25:30.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
11892023-12-05T05:25:30.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
11902023-12-05T05:25:30.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
11912023-12-05T05:25:30.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte0" rules = []
11922023-12-05T05:25:30.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:418 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
11932023-12-05T05:25:30.769ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 1250 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33825 req_id = 259d56d0-5909-46ee-aa4e-f63b175c7033 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
11942023-12-05T05:25:30.822ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11952023-12-05T05:25:31.134ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64951
11962023-12-05T05:25:31.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64951 req_id = c3b78ccd-0104-4a4d-b2d2-c80612662193 response_code = 200 uri = /network-bootstore-config
11972023-12-05T05:25:31.142ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:689
11982023-12-05T05:25:35.814ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
11992023-12-05T05:25:40.412ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34816
12002023-12-05T05:25:40.414ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:227 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: "localshark" }, nics: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, external_ips: [10.151.1.176], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], filter_hosts: Some([Vpc(Vni(15250817))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-2fa63a99737", read_only: false, slot: Slot(1), volume_construction_request: Volume { block_size: 512, id: 217839b6-510a-4dde-9884-022b7eabfb03, read_only_parent: Some(Url { block_size: 512, id: 1d741405-cbb9-4eea-9dae-e1c3b08cdf01, url: "http://[fd00:1122:3344:101::1]:54321/debian-11-genericcloud-amd64.raw" }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 217839b6-510a-4dde-9884-022b7eabfb03, key: Some("CnLfPm1XmIuQbLY0q1FheTfmYv1ooliIGj0t91YeZgM="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19000", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::17]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAlNCuFV4VXAAA0K4VXAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAJTQrhVeFVwAANCuFVwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI0OGQ0OGNlNC0wYmY0LTQ2NDMtOWVkNi0zZTc5MjkyYWRlMjciLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTGVDdjlodGN1M2M5UzFNN1dOZ0FNUkZXNzhoS3VaRFRsK2d1cVhxV01DcyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 instance_runtime = InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z } propolis_addr = [fd00:1122:3344:101::22]:12400 propolis_id = 07c89ad1-e69b-44f1-8b94-99bd96ad0694 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2023-12-05T05:25:40.015450Z }
12012023-12-05T05:25:40.414ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:261 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12022023-12-05T05:25:40.414ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:599 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 propolis_id = 07c89ad1-e69b-44f1-8b94-99bd96ad0694 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: "localshark" }, nics: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, external_ips: [10.151.1.176], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], filter_hosts: Some([Vpc(Vni(15250817))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7, kind: Instance { id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 53, 47])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(15250817), primary: true, slot: 1 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-2fa63a99737", read_only: false, slot: Slot(1), volume_construction_request: Volume { block_size: 512, id: 217839b6-510a-4dde-9884-022b7eabfb03, read_only_parent: Some(Url { block_size: 512, id: 1d741405-cbb9-4eea-9dae-e1c3b08cdf01, url: "http://[fd00:1122:3344:101::1]:54321/debian-11-genericcloud-amd64.raw" }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 217839b6-510a-4dde-9884-022b7eabfb03, key: Some("CnLfPm1XmIuQbLY0q1FheTfmYv1ooliIGj0t91YeZgM="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19000", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::17]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAlNCuFV4VXAAA0K4VXAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAJTQrhVeFVwAANCuFVwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI0OGQ0OGNlNC0wYmY0LTQ2NDMtOWVkNi0zZTc5MjkyYWRlMjciLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTGVDdjlodGN1M2M5UzFNN1dOZ0FNUkZXNzhoS3VaRFRsK2d1cVhxV01DcyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2023-12-05T05:25:40.015450Z }, propolis_addr: [fd00:1122:3344:101::22]:12400 }
12032023-12-05T05:25:40.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 1884 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34816 req_id = 4e1b92fe-3257-43fc-8090-ec4b51da6c10 response_code = 200 uri = /instances/48d48ce4-0bf4-4643-9ed6-3e79292ade27
12042023-12-05T05:25:40.515ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32849
12052023-12-05T05:25:40.534ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:383 port = Port { inner: PortInner { name: "opte6", _ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 247, 53, 47]), slot: 1, vni: Vni { inner: 15250817 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
12062023-12-05T05:25:40.534ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
12072023-12-05T05:25:40.567ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694 file = illumos-utils/src/zone.rs:314 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12082023-12-05T05:25:40.589ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694 file = illumos-utils/src/zone.rs:353 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12092023-12-05T05:25:40.812ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12102023-12-05T05:25:41.727ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46646
12112023-12-05T05:25:41.727ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 47 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46646 req_id = 9401e400-437e-4b59-9844-83a91ac8d490 response_code = 200 uri = /baseboard
12122023-12-05T05:25:42.079ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::22"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::c]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12132023-12-05T05:25:42.080ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:512 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12142023-12-05T05:25:42.259ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 41.338632ms file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12152023-12-05T05:25:42.308ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 135.749347ms file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12162023-12-05T05:25:42.459ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 157.605329ms file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12172023-12-05T05:25:42.628ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 545.112564ms file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12182023-12-05T05:25:43.187ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 905.156683ms file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12192023-12-05T05:25:44.102ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 1.158249481s file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12202023-12-05T05:25:45.270ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 754.590155ms file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12212023-12-05T05:25:45.813ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12222023-12-05T05:25:46.035ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 811.307853ms file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12232023-12-05T05:25:46.857ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694") failed: Property not found. retry in 1.401089531s file = illumos-utils/src/svc.rs:36 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 zone = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
12242023-12-05T05:25:48.288ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694 file = sled-agent/src/instance.rs:986 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12252023-12-05T05:25:48.296ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:995 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12262023-12-05T05:25:48.330ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1009 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12272023-12-05T05:25:48.331ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAlNCuFV4VXAAA0K4VXAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAJTQrhVeFVwAANCuFVwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI0OGQ0OGNlNC0wYmY0LTQ2NDMtOWVkNi0zZTc5MjkyYWRlMjciLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTGVDdjlodGN1M2M5UzFNN1dOZ0FNUkZXNzhoS3VaRFRsK2d1cVhxV01DcyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-2fa63a99737", read_only: false, slot: Slot(1), volume_construction_request: Volume { block_size: 512, id: 217839b6-510a-4dde-9884-022b7eabfb03, read_only_parent: Some(Url { block_size: 512, id: 1d741405-cbb9-4eea-9dae-e1c3b08cdf01, url: "http://[fd00:1122:3344:101::1]:54321/debian-11-genericcloud-amd64.raw" }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 217839b6-510a-4dde-9884-022b7eabfb03, key: Some("CnLfPm1XmIuQbLY0q1FheTfmYv1ooliIGj0t91YeZgM="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19000", "[fd00:1122:3344:101::16]:19000", "[fd00:1122:3344:101::17]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(1) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 48d48ce4-0bf4-4643-9ed6-3e79292ade27, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:450 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12282023-12-05T05:25:48.554ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:452 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12292023-12-05T05:25:48.554ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 8041322 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:32849 req_id = 9e55eca5-c5c0-4faf-b2a3-c3fbb0f4ec10 response_code = 200 uri = /instances/48d48ce4-0bf4-4643-9ed6-3e79292ade27/state
12302023-12-05T05:25:48.555ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2023-12-05T05:25:48.555069748Z } file = sled-agent/src/instance.rs:334 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12312023-12-05T05:25:48.555ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 new_instance_state = InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2023-12-05T05:25:48.555069748Z } propolis_id = 07c89ad1-e69b-44f1-8b94-99bd96ad0694
12322023-12-05T05:25:48.555ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z }, propolis_id: 07c89ad1-e69b-44f1-8b94-99bd96ad0694, vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2023-12-05T05:25:48.555069748Z } }
12332023-12-05T05:25:48.621ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2023-12-05T05:25:48.621921088Z } file = sled-agent/src/instance.rs:334 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12342023-12-05T05:25:48.621ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 new_instance_state = InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2023-12-05T05:25:48.621921088Z } propolis_id = 07c89ad1-e69b-44f1-8b94-99bd96ad0694
12352023-12-05T05:25:48.621ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z }, propolis_id: 07c89ad1-e69b-44f1-8b94-99bd96ad0694, vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2023-12-05T05:25:48.621921088Z } }
12362023-12-05T05:25:50.813ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12372023-12-05T05:25:52.482ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::d]:56770
12382023-12-05T05:25:52.484ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 1733 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56770 req_id = cb629b41-6364-47ba-9c0d-e75a8f9e8f06 response_code = 200 uri = /metrics/collect/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
12392023-12-05T05:25:55.813ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12402023-12-05T05:26:00.814ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12412023-12-05T05:26:05.815ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12422023-12-05T05:26:10.817ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12432023-12-05T05:26:11.756ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56968
12442023-12-05T05:26:11.756ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 41 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56968 req_id = 1dd71bb3-ad4b-47f8-8c7f-8338026ac849 response_code = 200 uri = /baseboard
12452023-12-05T05:26:15.816ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12462023-12-05T05:26:20.816ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12472023-12-05T05:26:22.477ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 325 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56770 req_id = 4ba8a19d-e384-4a77-898e-7f1acce236b0 response_code = 200 uri = /metrics/collect/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
12482023-12-05T05:26:25.816ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12492023-12-05T05:26:30.816ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12502023-12-05T05:26:35.818ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12512023-12-05T05:26:40.820ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12522023-12-05T05:26:41.786ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63907
12532023-12-05T05:26:41.786ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 49 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63907 req_id = 6adf7ce4-d4f6-4f15-b95c-7694e5e026cf response_code = 200 uri = /baseboard
12542023-12-05T05:26:45.823ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12552023-12-05T05:26:50.828ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12562023-12-05T05:26:52.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 226 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56770 req_id = ac852e1e-2554-4982-9ee1-846edf8e611c response_code = 200 uri = /metrics/collect/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
12572023-12-05T05:26:55.832ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12582023-12-05T05:27:00.835ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12592023-12-05T05:27:05.839ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12602023-12-05T05:27:10.843ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12612023-12-05T05:27:11.832ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56248
12622023-12-05T05:27:11.832ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 68 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56248 req_id = 248d854e-6b4d-4f7e-a28c-ea02bdb9dc40 response_code = 200 uri = /baseboard
12632023-12-05T05:27:15.847ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12642023-12-05T05:27:20.849ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12652023-12-05T05:27:22.491ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 208 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56770 req_id = c7a4aeb9-9e25-4ea8-8005-a83c52ee9014 response_code = 200 uri = /metrics/collect/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
12662023-12-05T05:27:25.850ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12672023-12-05T05:27:30.852ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12682023-12-05T05:27:35.853ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12692023-12-05T05:27:40.854ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12702023-12-05T05:27:41.868ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46955
12712023-12-05T05:27:41.868ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 32 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46955 req_id = 0802c8c1-865d-426c-b75e-235377c45730 response_code = 200 uri = /baseboard
12722023-12-05T05:27:45.856ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12732023-12-05T05:27:50.860ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12742023-12-05T05:27:52.494ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 270 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56770 req_id = 2c1c2f83-422b-4c76-aadf-ef49cef92dff response_code = 200 uri = /metrics/collect/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
12752023-12-05T05:27:55.862ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12762023-12-05T05:28:00.863ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12772023-12-05T05:28:05.864ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12782023-12-05T05:28:10.865ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12792023-12-05T05:28:11.905ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43150
12802023-12-05T05:28:11.906ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 32 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43150 req_id = 3c200a27-45d9-4415-976b-632658cf62ce response_code = 200 uri = /baseboard
12812023-12-05T05:28:15.867ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12822023-12-05T05:28:20.868ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12832023-12-05T05:28:22.494ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 241 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56770 req_id = a30045fb-364e-43d3-9877-41dde8807cc6 response_code = 200 uri = /metrics/collect/b21dc246-5dfb-4089-9a4d-5d9e5bd4a384
12842023-12-05T05:28:25.869ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
12852023-12-05T05:28:26.028ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:532
12862023-12-05T05:28:28.587ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47863
12872023-12-05T05:28:28.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 554 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47863 req_id = 6bfe0d98-0d5c-4cfc-92d7-8500e2f1b361 response_code = 200 uri = /instances/48d48ce4-0bf4-4643-9ed6-3e79292ade27/state
12882023-12-05T05:28:28.588ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2023-12-05T05:28:28.588375626Z } file = sled-agent/src/instance.rs:334 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12892023-12-05T05:28:28.588ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 new_instance_state = InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2023-12-05T05:28:28.588375626Z } propolis_id = 07c89ad1-e69b-44f1-8b94-99bd96ad0694
12902023-12-05T05:28:28.588ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z }, propolis_id: 07c89ad1-e69b-44f1-8b94-99bd96ad0694, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2023-12-05T05:28:28.588375626Z } }
12912023-12-05T05:28:28.610ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopped), migration_status: NoMigration, time: 2023-12-05T05:28:28.610469502Z } file = sled-agent/src/instance.rs:334 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12922023-12-05T05:28:28.610ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 new_instance_state = InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2023-12-05T05:28:28.610469502Z } propolis_id = 07c89ad1-e69b-44f1-8b94-99bd96ad0694
12932023-12-05T05:28:28.610ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(07c89ad1-e69b-44f1-8b94-99bd96ad0694), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2023-12-05T05:25:40.069703709Z }, propolis_id: 07c89ad1-e69b-44f1-8b94-99bd96ad0694, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2023-12-05T05:28:28.610469502Z } }
12942023-12-05T05:28:28.777ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2023-12-05T05:28:28.777447381Z } file = sled-agent/src/instance.rs:334 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12952023-12-05T05:28:28.777ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:356 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2023-12-05T05:28:28.777447381Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2023-12-05T05:28:28.777447381Z } propolis_id = 07c89ad1-e69b-44f1-8b94-99bd96ad0694
12962023-12-05T05:28:28.777ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:371 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
12972023-12-05T05:28:28.777ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
12982023-12-05T05:28:28.778ZINFOSledAgent (StorageManager): Received GetLatestResources(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: true, is_tx_task_set: false } }) }) file = sled-storage/src/manager.rs:346
12992023-12-05T05:28:28.778ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/a462a7f7-b628-40fe-80ff-4e4189e2d62b/debug/bundle/zone", "/pool/int/b462a7f7-b628-40fe-80ff-4e4189e2d62b/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug", "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug", "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/debug", "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/debug", "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/debug", "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/debug", "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/debug", "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug", "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:436 zone_name = oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694
13002023-12-05T05:28:29.424ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1229 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694", bundle_id: d9d44915-3c56-41e1-92e9-cd77f2a87c6c }, time_created: 2023-12-05T05:28:28.778340222Z, version: 0, cause: TerminatedInstance }
13012023-12-05T05:28:29.424ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_07c89ad1-e69b-44f1-8b94-99bd96ad0694 file = sled-agent/src/instance.rs:544 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
13022023-12-05T05:28:30.870ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}
13032023-12-05T05:28:31.011ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:268 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
13042023-12-05T05:28:31.036ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:259 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2023-12-05T05:28:28.777447381Z }, propolis_id: 07c89ad1-e69b-44f1-8b94-99bd96ad0694, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2023-12-05T05:28:28.777447381Z } }
13052023-12-05T05:28:31.107ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:769 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62402
13062023-12-05T05:28:31.108ZWARNSledAgent (PortManager): unset_virtual_nic_host unimplmented file = illumos-utils/src/opte/port_manager.rs:494
13072023-12-05T05:28:31.108ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:853 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:62402 req_id = a7606500-78ca-468c-b485-4a44d0cebb89 response_code = 204 uri = /v2p/4245bc7d-a79b-4fe9-8ca3-c2cadd7d95d7
13082023-12-05T05:28:31.226ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:487 instance_id = 48d48ce4-0bf4-4643-9ed6-3e79292ade27
13092023-12-05T05:28:35.872ZINFOSledAgent (bootstore_ddmd_poller): Received prefixes from ddmd DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:119 prefixes = {"fe80::90f4:54ff:fe46:1e98": [PathVector { destination: Ipv6Prefix { addr: fd00:99::, len: 64 }, path: ["oxz_switch"] }]}