Max level shown:
1 [ Jan 31 05:57:59 Disabled. ]
2 [ Jan 31 05:57:59 Rereading configuration. ]
3 [ Jan 31 05:58:00 Enabled. ]
4 [ Jan 31 05:58:00 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
5 [ Jan 31 05:58:00 Method "start" exited with status 0. ]
6 [ Jan 31 05:58:00 Rereading configuration. ]
7 [ Jan 31 05:58:00 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 212"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=212
11 + getopts c: o
12 + [[ -n 212 ]]
13 + pgrep -c 212
14 + PARENT_PID=2276
15 + [[ -n 2276 ]]
16 + pgrep -P 2276
17 + CHILD_PID=2277
18 + [[ -n 2277 ]]
19 + kill -USR1 2277
20 [ Jan 31 05:58:00 Method "refresh" exited with status 0. ]
21 [ Jan 31 05:58:00 Stopping because all processes in service exited. ]
22 [ Jan 31 05:58:00 Executing stop method (:kill). ]
23 [ Jan 31 05:58:00 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
24 [ Jan 31 05:58:00 Method "start" exited with status 0. ]
252026-01-31T05:58:00.381ZINFOslog-rs (mgd): signal handler waiting for context module = admin unit = signal
262026-01-31T05:58:00.381ZINFOslog-rs (bgp): dispatcher started listen_address = [::]:179 module = neighbor unit = dispatcher
272026-01-31T05:58:00.381ZDEBGslog-rs (bgp): listener bind: [::]:179 listen_address = [::]:179 module = neighbor unit = dispatcher
282026-01-31T05:58:00.402ZINFOslog-rs (mgd): signal handler got context module = admin unit = signal
292026-01-31T05:58:00.402ZINFOslog-rs (mgd): starting bgp routers: {} module = admin unit = daemon
302026-01-31T05:58:00.402ZINFOslog-rs (mgd): starting bfd sessions: [] module = admin unit = daemon
312026-01-31T05:58:00.403ZERROslog-rs (mgd): failed to start stats server: underlay address not found error = underlay address not found module = admin params = hostname oxz_switch, rack bcc85926-2010-4bb0-8bd4-4de8280fd651, sled 77b517e2-2db4-4507-b73d-8b24f0e28fb4 unit = daemon
322026-01-31T05:58:00.403ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
332026-01-31T05:58:00.404ZDEBGslog-rs: client response result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:12225, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }) unit = gateway-client
342026-01-31T05:58:00.404ZERROslog-rs: failed to resolve switch slot error = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id)
352026-01-31T05:58:00.404ZINFOslog-rs: listening on [::]:4676
362026-01-31T05:58:00.404ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
372026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
382026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bfd/peers unit = api_server versions = all
392026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bfd/peers unit = api_server versions = all
402026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bfd/peers/{addr} unit = api_server versions = all
412026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/clear/neighbor unit = api_server versions = all
422026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/checker unit = api_server versions = all
432026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/checker unit = api_server versions = all
442026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/checker unit = api_server versions = all
452026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/checker unit = api_server versions = all
462026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/neighbor unit = api_server versions = all
472026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbor unit = api_server versions = all
482026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/neighbor unit = api_server versions = all
492026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/neighbor unit = api_server versions = all
502026-01-31T05:58:00.405ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
512026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbors unit = api_server versions = all
522026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin4 unit = api_server versions = all
532026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin4 unit = api_server versions = all
542026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin4 unit = api_server versions = all
552026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin4 unit = api_server versions = all
562026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
572026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
582026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
592026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
602026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/router unit = api_server versions = all
612026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/router unit = api_server versions = all
622026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/router unit = api_server versions = all
632026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/router unit = api_server versions = all
642026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/routers unit = api_server versions = all
652026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/shaper unit = api_server versions = all
662026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/shaper unit = api_server versions = all
672026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/shaper unit = api_server versions = all
682026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/shaper unit = api_server versions = all
692026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/fsm unit = api_server versions = all starting from 2.0.0
702026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/message unit = api_server versions = all starting from 2.0.0
712026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/message-history unit = api_server versions = all ending with 2.0.0
722026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/omicron/apply unit = api_server versions = all
732026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/exported unit = api_server versions = all
742026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/imported unit = api_server versions = all ending with 2.0.0
752026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all ending with 2.0.0
762026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all starting from 2.0.0
772026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/selected unit = api_server versions = all ending with 2.0.0
782026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
792026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
802026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/imported unit = api_server versions = all starting from 2.0.0
812026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/selected unit = api_server versions = all starting from 2.0.0
822026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route4 unit = api_server versions = all
832026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route4 unit = api_server versions = all
842026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route4 unit = api_server versions = all
852026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
862026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
872026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
882026-01-31T05:58:00.405ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /switch/identifiers unit = api_server versions = all starting from 3.0.0
892026-01-31T05:58:00.405ZINFOslog-rs (mgd): listening local_addr = [::]:4676 module = admin unit = api_server
902026-01-31T05:58:00.405ZDEBGslog-rs (mgd): successfully registered DTrace USDT probes local_addr = [::]:4676 module = admin unit = api_server
912026-01-31T05:58:00.405ZINFOslog-rs: admin: server started
922026-01-31T05:58:01.405ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
932026-01-31T05:58:02.405ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
942026-01-31T05:58:03.406ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
952026-01-31T05:58:04.406ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
962026-01-31T05:58:05.406ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
972026-01-31T05:58:06.407ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
982026-01-31T05:58:07.407ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
992026-01-31T05:58:08.407ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
1002026-01-31T05:58:09.408ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
1012026-01-31T05:58:10.405ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
1022026-01-31T05:58:10.405ZDEBGslog-rs: client response result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:12225, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }) unit = gateway-client
1032026-01-31T05:58:10.405ZERROslog-rs: failed to resolve switch slot error = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id)
1042026-01-31T05:58:10.408ZWARNslog-rs (mg-lower): failed to ensure TEP address fdf5:f188:bf1f:eda::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fdf5:f188:bf1f:eda::1 unit = dpd
105 [ Jan 31 05:58:11 Rereading configuration. ]
106 [ Jan 31 05:58:11 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 218"). ]
107 + CONTRACT=''
108 + getopts c: o
109 + CONTRACT=218
110 + getopts c: o
111 + [[ -n 218 ]]
112 + pgrep -c 218
113 + PARENT_PID=2306
114 + [[ -n 2306 ]]
115 + pgrep -P 2306
116 + CHILD_PID=2308
117 + [[ -n 2308 ]]
118 + kill -USR1 2308
1192026-01-31T05:58:11.786ZINFOslog-rs (mgd): handling smf refresh module = admin unit = smf
120 [ Jan 31 05:58:11 Method "refresh" exited with status 0. ]
1212026-01-31T05:58:11.788ZINFOslog-rs (mgd): starting stats server on smf refresh module = admin unit = smf
1222026-01-31T05:58:11.788ZINFOslog-rs (mgd): started stats server on smf refresh module = admin unit = smf
123 Jan 31 05:58:11.789 DEBG registered DTrace probes
124 Jan 31 05:58:11.789 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
125 Jan 31 05:58:11.789 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.6/src/server.rs:328
126 Jan 31 05:58:11.789 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
127 Jan 31 05:58:11.789 DEBG Nexus IP not provided, will use DNS to resolve it
128 Jan 31 05:58:11.789 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/internal-dns/resolver/src/resolver.rs:144
129 Jan 31 05:58:11.790 DEBG starting producer registration task
130 Jan 31 05:58:11.790 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: b3d51edc-54da-440e-beb3-648c610dfa94, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:279
131 Jan 31 05:58:11.790 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
1322026-01-31T05:58:12.204ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::1]:54860 unit = api_server
1332026-01-31T05:58:12.204ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:54860 req_id = 0b45a96a-bffa-4530-983a-94aef92c2035 unit = api_server uri = /static/route4 version = 3.0.0
1342026-01-31T05:58:12.205ZINFOslog-rs (mgd): request completed latency_us = 576 local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:54860 req_id = 0b45a96a-bffa-4530-983a-94aef92c2035 response_code = 204 unit = api_server uri = /static/route4
1352026-01-31T05:58:12.206ZERROslog-rs (mg-lower): handling change failed: libnet error route io error No such process (os error 3) error = libnet error route io error No such process (os error 3) module = sync unit = event_loop
1362026-01-31T05:58:12.206ZINFOslog-rs (mg-lower): restarting sync loop module = sync unit = event_loop
137 Jan 31 05:58:16.793 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("296d1cab-b1ce-4146-9e91-96b0145b5ac3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95644442-9bd0-4b35-b333-f2f39dee5b9c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 671, tv_nsec: 442945354 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
138 Jan 31 05:58:16.793 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
139 Jan 31 05:58:16.793 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1402026-01-31T05:58:20.406ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
1412026-01-31T05:58:20.406ZDEBGslog-rs: client response result = Ok(Response { url: "http://[::1]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "42a5b560-b756-457b-b5b6-4022500fd213", "content-length": "26", "date": "Sat, 31 Jan 2026 05:58:20 GMT"} }) unit = gateway-client
1422026-01-31T05:58:20.406ZINFOslog-rs: we are in switch slot 0
143 Jan 31 05:58:31.781 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
144 Jan 31 05:58:31.781 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 20.004749789s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
145 Jan 31 05:58:31.988 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("296d1cab-b1ce-4146-9e91-96b0145b5ac3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95644442-9bd0-4b35-b333-f2f39dee5b9c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 686, tv_nsec: 652889613 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
146 Jan 31 05:58:31.988 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
147 Jan 31 05:58:31.988 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
148 Jan 31 05:58:46.950 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
149 Jan 31 05:58:46.950 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 35.215569837s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
150 Jan 31 05:58:47.597 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("296d1cab-b1ce-4146-9e91-96b0145b5ac3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95644442-9bd0-4b35-b333-f2f39dee5b9c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 702, tv_nsec: 302992843 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
151 Jan 31 05:58:47.597 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
152 Jan 31 05:58:47.598 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
153 Jan 31 05:59:02.597 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
154 Jan 31 05:59:02.598 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 50.873593069s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
155 Jan 31 05:59:03.530 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("296d1cab-b1ce-4146-9e91-96b0145b5ac3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95644442-9bd0-4b35-b333-f2f39dee5b9c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 718, tv_nsec: 245406105 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
156 Jan 31 05:59:03.531 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
157 Jan 31 05:59:03.533 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
158 Jan 31 05:59:18.534 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
159 Jan 31 05:59:18.534 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 66.809150413s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
160 Jan 31 05:59:20.780 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("296d1cab-b1ce-4146-9e91-96b0145b5ac3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95644442-9bd0-4b35-b333-f2f39dee5b9c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 735, tv_nsec: 495631420 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
161 Jan 31 05:59:20.781 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
162 Jan 31 05:59:20.781 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
163 Jan 31 05:59:35.782 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
164 Jan 31 05:59:35.782 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 84.057717361s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
165 Jan 31 05:59:38.492 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("296d1cab-b1ce-4146-9e91-96b0145b5ac3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95644442-9bd0-4b35-b333-f2f39dee5b9c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 753, tv_nsec: 207454067 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
166 Jan 31 05:59:38.492 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
167 Jan 31 05:59:38.492 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
168 Jan 31 05:59:53.493 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
169 Jan 31 05:59:53.493 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)", delay: 101.768234118s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79b29e0/oximeter/producer/src/lib.rs:371
170 Jan 31 06:00:00.999 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("296d1cab-b1ce-4146-9e91-96b0145b5ac3.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("95644442-9bd0-4b35-b333-f2f39dee5b9c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d9dc22c1-095b-4f7e-a8d4-02c4df68f8ee.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 775, tv_nsec: 714521413 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
171 Jan 31 06:00:01.000 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
172 Jan 31 06:00:01.000 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task