Max level shown:
1 [ Feb 4 23:27:50 Disabled. ]
2 [ Feb 4 23:27:50 Rereading configuration. ]
3 [ Feb 4 23:27:51 Enabled. ]
4 [ Feb 4 23:27:51 Rereading configuration. ]
5 [ Feb 4 23:27:51 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
6 [ Feb 4 23:27:51 Method "start" exited with status 0. ]
72026-02-04T23:27:51.478ZINFOslog-rs (mgd): signal handler waiting for context module = admin unit = signal
82026-02-04T23:27:51.479ZINFOslog-rs (bgp): dispatcher started listen_address = [::]:179 module = neighbor unit = dispatcher
92026-02-04T23:27:51.479ZDEBGslog-rs (bgp): listener bind: [::]:179 listen_address = [::]:179 module = neighbor unit = dispatcher
102026-02-04T23:27:51.500ZINFOslog-rs (mgd): signal handler got context module = admin unit = signal
112026-02-04T23:27:51.500ZINFOslog-rs (mgd): starting bgp routers: {} module = admin unit = daemon
122026-02-04T23:27:51.500ZINFOslog-rs (mgd): starting bfd sessions: [] module = admin unit = daemon
132026-02-04T23:27:51.501ZERROslog-rs (mgd): failed to start stats server: underlay address not found error = underlay address not found module = admin params = hostname oxz_switch, rack dec67623-891e-4f97-8d2b-9296596bf9a7, sled 811afa9f-25f5-4cac-89bc-985b2699dbf5 unit = daemon
142026-02-04T23:27:51.502ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
152026-02-04T23:27:51.502ZDEBGslog-rs: client response result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:12225, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }) unit = gateway-client
162026-02-04T23:27:51.502ZERROslog-rs: failed to resolve switch slot error = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id)
172026-02-04T23:27:51.503ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
182026-02-04T23:27:51.503ZINFOslog-rs: listening on [::]:4676
192026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
202026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
212026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bfd/peers unit = api_server versions = all
222026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bfd/peers unit = api_server versions = all
232026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bfd/peers/{addr} unit = api_server versions = all
242026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/clear/neighbor unit = api_server versions = all
252026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/checker unit = api_server versions = all
262026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/checker unit = api_server versions = all
272026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/checker unit = api_server versions = all
282026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/checker unit = api_server versions = all
292026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/neighbor unit = api_server versions = all
302026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbor unit = api_server versions = all
312026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/neighbor unit = api_server versions = all
322026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/neighbor unit = api_server versions = all
332026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbors unit = api_server versions = all
342026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin4 unit = api_server versions = all
352026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin4 unit = api_server versions = all
362026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin4 unit = api_server versions = all
372026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin4 unit = api_server versions = all
382026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
392026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
402026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
412026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
422026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/router unit = api_server versions = all
432026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/router unit = api_server versions = all
442026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/router unit = api_server versions = all
452026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/router unit = api_server versions = all
462026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/routers unit = api_server versions = all
472026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/shaper unit = api_server versions = all
482026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/shaper unit = api_server versions = all
492026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/shaper unit = api_server versions = all
502026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/shaper unit = api_server versions = all
512026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/fsm unit = api_server versions = all starting from 2.0.0
522026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/message unit = api_server versions = all starting from 2.0.0
532026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/message-history unit = api_server versions = all ending with 2.0.0
542026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/omicron/apply unit = api_server versions = all
552026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/exported unit = api_server versions = all
562026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/imported unit = api_server versions = all ending with 2.0.0
572026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all ending with 2.0.0
582026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all starting from 2.0.0
592026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/selected unit = api_server versions = all ending with 2.0.0
602026-02-04T23:27:51.503ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
612026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
622026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/imported unit = api_server versions = all starting from 2.0.0
632026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/selected unit = api_server versions = all starting from 2.0.0
642026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route4 unit = api_server versions = all
652026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route4 unit = api_server versions = all
662026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route4 unit = api_server versions = all
672026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
682026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
692026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
702026-02-04T23:27:51.504ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /switch/identifiers unit = api_server versions = all starting from 3.0.0
712026-02-04T23:27:51.504ZINFOslog-rs (mgd): listening local_addr = [::]:4676 module = admin unit = api_server
722026-02-04T23:27:51.504ZDEBGslog-rs (mgd): successfully registered DTrace USDT probes local_addr = [::]:4676 module = admin unit = api_server
732026-02-04T23:27:51.504ZINFOslog-rs: admin: server started
742026-02-04T23:27:52.504ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
752026-02-04T23:27:53.504ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
762026-02-04T23:27:54.504ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
772026-02-04T23:27:55.505ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
782026-02-04T23:27:56.505ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
792026-02-04T23:27:57.533ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
802026-02-04T23:27:58.527ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
812026-02-04T23:27:59.528ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
822026-02-04T23:28:00.528ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
832026-02-04T23:28:01.504ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
842026-02-04T23:28:01.504ZDEBGslog-rs: client response result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:12225, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }) unit = gateway-client
852026-02-04T23:28:01.504ZERROslog-rs: failed to resolve switch slot error = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id)
862026-02-04T23:28:01.528ZWARNslog-rs (mg-lower): failed to ensure TEP address fd65:eee0:5e33:99ce::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fd65:eee0:5e33:99ce::1 unit = dpd
87 [ Feb 4 23:28:02 Rereading configuration. ]
88 [ Feb 4 23:28:02 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 211"). ]
89 + CONTRACT=''
90 + getopts c: o
91 + CONTRACT=211
92 + getopts c: o
93 + [[ -n 211 ]]
94 + pgrep -c 211
95 + PARENT_PID=2281
96 + [[ -n 2281 ]]
97 + pgrep -P 2281
98 + CHILD_PID=2282
99 + [[ -n 2282 ]]
100 + kill -USR1 2282
1012026-02-04T23:28:02.874ZINFOslog-rs (mgd): handling smf refresh module = admin unit = smf
1022026-02-04T23:28:02.876ZINFOslog-rs (mgd): starting stats server on smf refresh module = admin unit = smf
103 [ Feb 4 23:28:02 Method "refresh" exited with status 0. ]
1042026-02-04T23:28:02.876ZINFOslog-rs (mgd): started stats server on smf refresh module = admin unit = smf
105 Feb 04 23:28:02.877 DEBG registered DTrace probes
106 Feb 04 23:28:02.877 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
107 Feb 04 23:28:02.877 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:328
108 Feb 04 23:28:02.878 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
109 Feb 04 23:28:02.878 DEBG Nexus IP not provided, will use DNS to resolve it
110 Feb 04 23:28:02.878 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/internal-dns/resolver/src/resolver.rs:144
111 Feb 04 23:28:02.878 DEBG starting producer registration task
112 Feb 04 23:28:02.878 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 894775a5-bfba-4e1e-afb7-5506800c1e24, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:279
113 Feb 04 23:28:02.878 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
1142026-02-04T23:28:03.248ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::1]:60131 unit = api_server
1152026-02-04T23:28:03.248ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:60131 req_id = ec6b83cc-8829-490f-ba50-3ff4e08023c9 unit = api_server uri = /static/route4 version = 3.0.0
1162026-02-04T23:28:03.249ZINFOslog-rs (mgd): request completed latency_us = 614 local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:60131 req_id = ec6b83cc-8829-490f-ba50-3ff4e08023c9 response_code = 204 unit = api_server uri = /static/route4
1172026-02-04T23:28:03.250ZERROslog-rs (mg-lower): handling change failed: libnet error route io error No such process (os error 3) error = libnet error route io error No such process (os error 3) module = sync unit = event_loop
1182026-02-04T23:28:03.250ZINFOslog-rs (mg-lower): restarting sync loop module = sync unit = event_loop
119 Feb 04 23:28:07.881 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1756efb2-48e3-4a81-b731-4d0a0c88feb9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("60ce65ca-1d2b-4f5a-9afa-9d25546fdef2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1663, tv_nsec: 671106946 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
120 Feb 04 23:28:07.881 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
121 Feb 04 23:28:07.882 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1222026-02-04T23:28:11.505ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
1232026-02-04T23:28:11.506ZDEBGslog-rs: client response result = Ok(Response { url: "http://[::1]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "2faa0796-b0a1-4566-90d7-3ea2d4de61c2", "content-length": "26", "date": "Wed, 04 Feb 2026 23:28:11 GMT"} }) unit = gateway-client
1242026-02-04T23:28:11.506ZINFOslog-rs: we are in switch slot 0
125 Feb 04 23:28:22.882 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
126 Feb 04 23:28:22.882 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 20.004252639s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
127 Feb 04 23:28:23.197 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1756efb2-48e3-4a81-b731-4d0a0c88feb9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("60ce65ca-1d2b-4f5a-9afa-9d25546fdef2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1678, tv_nsec: 987377483 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
128 Feb 04 23:28:23.198 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
129 Feb 04 23:28:23.198 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
130 Feb 04 23:28:38.205 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
131 Feb 04 23:28:38.205 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 35.321329674s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
132 Feb 04 23:28:38.751 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1756efb2-48e3-4a81-b731-4d0a0c88feb9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("60ce65ca-1d2b-4f5a-9afa-9d25546fdef2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1694, tv_nsec: 535509115 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
133 Feb 04 23:28:38.751 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
134 Feb 04 23:28:38.751 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
135 Feb 04 23:28:53.763 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
136 Feb 04 23:28:53.765 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 50.87923068s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
137 Feb 04 23:28:54.295 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1756efb2-48e3-4a81-b731-4d0a0c88feb9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("60ce65ca-1d2b-4f5a-9afa-9d25546fdef2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1710, tv_nsec: 77394497 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
138 Feb 04 23:28:54.297 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
139 Feb 04 23:28:54.300 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
140 Feb 04 23:29:09.303 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
141 Feb 04 23:29:09.305 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 66.420729717s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
142 Feb 04 23:29:12.286 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1756efb2-48e3-4a81-b731-4d0a0c88feb9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("60ce65ca-1d2b-4f5a-9afa-9d25546fdef2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1728, tv_nsec: 70419996 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
143 Feb 04 23:29:12.320 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
144 Feb 04 23:29:12.322 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
145 Feb 04 23:29:27.322 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
146 Feb 04 23:29:27.322 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 84.438763155s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
147 Feb 04 23:29:31.191 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1756efb2-48e3-4a81-b731-4d0a0c88feb9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("60ce65ca-1d2b-4f5a-9afa-9d25546fdef2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1746, tv_nsec: 975257292 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
148 Feb 04 23:29:31.191 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
149 Feb 04 23:29:31.191 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
150 Feb 04 23:29:35.727 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::c]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
151 Feb 04 23:29:35.727 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 92.843342756s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
152 Feb 04 23:29:42.690 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1756efb2-48e3-4a81-b731-4d0a0c88feb9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("60ce65ca-1d2b-4f5a-9afa-9d25546fdef2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("df4483de-4b3e-4457-be14-7464a95b3f5f.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 1758, tv_nsec: 474829265 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
153 Feb 04 23:29:42.691 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
154 Feb 04 23:29:42.691 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
155 Feb 04 23:29:45.525 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "a4b26330-6e3c-4b43-a3de-3b2e013ef2dd", "content-length": "41", "date": "Wed, 04 Feb 2026 23:29:45 GMT"} }), component: producer-registration-task
156 Feb 04 23:29:45.526 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
1572026-02-04T23:29:49.134ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::c]:46303 unit = api_server
1582026-02-04T23:29:49.141ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:46303 req_id = 38ab6c14-cc81-41ae-bf28-558f53aa0260 unit = api_server uri = /bfd/peers version = 3.0.0
1592026-02-04T23:29:49.148ZINFOslog-rs (mgd): request completed latency_us = 10024 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:46303 req_id = 38ab6c14-cc81-41ae-bf28-558f53aa0260 response_code = 200 unit = api_server uri = /bfd/peers
1602026-02-04T23:29:50.250ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::b]:45140 unit = api_server
1612026-02-04T23:29:50.250ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:45140 req_id = 46a10439-991e-4df2-9fcb-b1dba8407158 unit = api_server uri = /bfd/peers version = 3.0.0
1622026-02-04T23:29:50.250ZINFOslog-rs (mgd): request completed latency_us = 78 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:45140 req_id = 46a10439-991e-4df2-9fcb-b1dba8407158 response_code = 200 unit = api_server uri = /bfd/peers
1632026-02-04T23:29:50.263ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::a]:52722 unit = api_server
1642026-02-04T23:29:50.263ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:52722 req_id = e3826e87-1faa-4998-8232-e1dd1231bd4f unit = api_server uri = /bfd/peers version = 3.0.0
1652026-02-04T23:29:50.264ZINFOslog-rs (mgd): request completed latency_us = 83 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:52722 req_id = e3826e87-1faa-4998-8232-e1dd1231bd4f response_code = 200 unit = api_server uri = /bfd/peers
166 Feb 04 23:29:58.643 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:42010, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:1025
167 Feb 04 23:29:58.657 INFO request completed, latency_us: 12970, response_code: 200, uri: /894775a5-bfba-4e1e-afb7-5506800c1e24, method: GET, req_id: 73e41e81-55d0-43e8-ab0c-3abc814711f8, remote_addr: [fd00:1122:3344:101::d]:42010, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867
168 Feb 04 23:29:59.295 INFO request completed, latency_us: 334, response_code: 200, uri: /894775a5-bfba-4e1e-afb7-5506800c1e24, method: GET, req_id: 87d98093-f4cc-4fe1-b72a-e5ef616abf3c, remote_addr: [fd00:1122:3344:101::d]:42010, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867
169 Feb 04 23:30:00.295 INFO request completed, latency_us: 536, response_code: 200, uri: /894775a5-bfba-4e1e-afb7-5506800c1e24, method: GET, req_id: 4bc42d31-f1c7-4b31-9872-b99b0421464f, remote_addr: [fd00:1122:3344:101::d]:42010, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867
170 Feb 04 23:30:01.295 INFO request completed, latency_us: 367, response_code: 200, uri: /894775a5-bfba-4e1e-afb7-5506800c1e24, method: GET, req_id: d154e5aa-27a2-4ed7-a749-46acc7511da2, remote_addr: [fd00:1122:3344:101::d]:42010, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:867