Max level shown:
1 [ Mar 24 17:53:31 Disabled. ]
2 [ Mar 24 17:53:31 Rereading configuration. ]
3 [ Mar 24 17:53:32 Enabled. ]
4 [ Mar 24 17:53:32 Rereading configuration. ]
5 [ Mar 24 17:53:32 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
6 [ Mar 24 17:53:32 Method "start" exited with status 0. ]
72026-03-24T17:53:32.144ZINFOslog-rs (mgd): signal handler waiting for context module = admin unit = signal
82026-03-24T17:53:32.166ZINFOslog-rs (mgd): interface monitor started module = unnumbered manager unit = daemon
92026-03-24T17:53:32.166ZINFOslog-rs (bgp): dispatcher started listen_address = [::]:179 module = neighbor unit = dispatcher
102026-03-24T17:53:32.166ZDEBGslog-rs (bgp): listener bind: [::]:179 listen_address = [::]:179 module = neighbor unit = dispatcher
112026-03-24T17:53:32.167ZINFOslog-rs (mgd): signal handler got context module = admin unit = signal
122026-03-24T17:53:32.168ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
132026-03-24T17:53:32.168ZDEBGslog-rs: client response result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:12225, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }) unit = gateway-client
142026-03-24T17:53:32.168ZERROslog-rs: failed to resolve switch slot error = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id)
152026-03-24T17:53:32.169ZINFOslog-rs (mgd): starting bgp routers: {} module = admin unit = daemon
162026-03-24T17:53:32.170ZINFOslog-rs (mgd): starting bfd sessions: [] module = admin unit = daemon
172026-03-24T17:53:32.170ZERROslog-rs (mgd): failed to start stats server: underlay address not found error = underlay address not found module = admin params = hostname oxz_switch, rack ed284a5c-01fe-43b0-9594-204a9e898855, sled e0acdd00-0cab-40d6-94fa-f114946d3aa0 unit = daemon
182026-03-24T17:53:32.171ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
192026-03-24T17:53:32.172ZINFOslog-rs: listening on [::]:4676
202026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
212026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
222026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bfd/peers unit = api_server versions = all
232026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bfd/peers unit = api_server versions = all
242026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bfd/peers/{addr} unit = api_server versions = all
252026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/clear/neighbor unit = api_server versions = all ending with 4.0.0
262026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/clear/neighbor unit = api_server versions = all starting from 4.0.0
272026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/clear/unnumbered-neighbor unit = api_server versions = all starting from 5.0.0
282026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/checker unit = api_server versions = all
292026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/checker unit = api_server versions = all
302026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/checker unit = api_server versions = all
312026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/checker unit = api_server versions = all
322026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/neighbor unit = api_server versions = all ending with 4.0.0
332026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/neighbor unit = api_server versions = from 4.0.0 to 5.0.0
342026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbor unit = api_server versions = all ending with 4.0.0
352026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbor unit = api_server versions = from 4.0.0 to 5.0.0
362026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/neighbor unit = api_server versions = all ending with 4.0.0
372026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/neighbor unit = api_server versions = from 4.0.0 to 5.0.0
382026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/neighbor unit = api_server versions = all starting from 5.0.0
392026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/neighbor unit = api_server versions = all ending with 4.0.0
402026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/neighbor unit = api_server versions = from 4.0.0 to 5.0.0
412026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/neighbor unit = api_server versions = all starting from 5.0.0
422026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/neighbor/{asn}/{peer} unit = api_server versions = all starting from 5.0.0
432026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbor/{asn}/{peer} unit = api_server versions = all starting from 5.0.0
442026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbors unit = api_server versions = all ending with 4.0.0
452026-03-24T17:53:32.172ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbors unit = api_server versions = from 4.0.0 to 5.0.0
462026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbors/{asn} unit = api_server versions = all starting from 5.0.0
472026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin4 unit = api_server versions = all
482026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin4 unit = api_server versions = all
492026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin4 unit = api_server versions = all
502026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin4 unit = api_server versions = all
512026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
522026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
532026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
542026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
552026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/router unit = api_server versions = all
562026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/router unit = api_server versions = all
572026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/router unit = api_server versions = all
582026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/router unit = api_server versions = all
592026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/routers unit = api_server versions = all
602026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/shaper unit = api_server versions = all
612026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/shaper unit = api_server versions = all
622026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/shaper unit = api_server versions = all
632026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/shaper unit = api_server versions = all
642026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/unnumbered-neighbor unit = api_server versions = all starting from 5.0.0
652026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/unnumbered-neighbor unit = api_server versions = all starting from 5.0.0
662026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/unnumbered-neighbor unit = api_server versions = all starting from 5.0.0
672026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/unnumbered-neighbor unit = api_server versions = all starting from 5.0.0
682026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/unnumbered-neighbors unit = api_server versions = all starting from 5.0.0
692026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/fsm unit = api_server versions = from 2.0.0 to 5.0.0
702026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/fsm unit = api_server versions = all starting from 5.0.0
712026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/message unit = api_server versions = from 2.0.0 to 5.0.0
722026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/message unit = api_server versions = all starting from 5.0.0
732026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/message-history unit = api_server versions = all ending with 2.0.0
742026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/omicron/apply unit = api_server versions = all ending with 4.0.0
752026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/omicron/apply unit = api_server versions = all starting from 4.0.0
762026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/exported unit = api_server versions = all ending with 5.0.0
772026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/exported unit = api_server versions = from 5.0.0 to 6.0.0
782026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/exported unit = api_server versions = all starting from 6.0.0
792026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/imported unit = api_server versions = all ending with 2.0.0
802026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all ending with 2.0.0
812026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = from 2.0.0 to 4.0.0
822026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = from 4.0.0 to 5.0.0
832026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all starting from 5.0.0
842026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/selected unit = api_server versions = all ending with 2.0.0
852026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /ndp/interface unit = api_server versions = all starting from 5.0.0
862026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /ndp/interfaces unit = api_server versions = all starting from 5.0.0
872026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /ndp/manager unit = api_server versions = all starting from 5.0.0
882026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
892026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
902026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/imported unit = api_server versions = from 2.0.0 to 5.0.0
912026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/imported unit = api_server versions = all starting from 5.0.0
922026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/selected unit = api_server versions = from 2.0.0 to 5.0.0
932026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/selected unit = api_server versions = all starting from 5.0.0
942026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route4 unit = api_server versions = all
952026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route4 unit = api_server versions = all
962026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route4 unit = api_server versions = all
972026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
982026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
992026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
1002026-03-24T17:53:32.173ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /switch/identifiers unit = api_server versions = all starting from 3.0.0
1012026-03-24T17:53:32.173ZINFOslog-rs (mgd): listening local_addr = [::]:4676 module = admin unit = api_server
1022026-03-24T17:53:32.173ZDEBGslog-rs (mgd): successfully registered DTrace USDT probes local_addr = [::]:4676 module = admin unit = api_server
1032026-03-24T17:53:32.173ZINFOslog-rs: admin: server started
1042026-03-24T17:53:33.171ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1052026-03-24T17:53:34.172ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1062026-03-24T17:53:35.172ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1072026-03-24T17:53:36.172ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1082026-03-24T17:53:37.173ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1092026-03-24T17:53:38.173ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1102026-03-24T17:53:39.173ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1112026-03-24T17:53:40.174ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1122026-03-24T17:53:41.174ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
1132026-03-24T17:53:42.169ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
1142026-03-24T17:53:42.170ZDEBGslog-rs: client response result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [::1]:12225, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }) unit = gateway-client
1152026-03-24T17:53:42.170ZERROslog-rs: failed to resolve switch slot error = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id)
1162026-03-24T17:53:42.175ZWARNslog-rs (mg-lower): failed to ensure TEP address fda7:4b94:a27:6543::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fda7:4b94:a27:6543::1 unit = dpd
117 [ Mar 24 17:53:43 Rereading configuration. ]
118 [ Mar 24 17:53:43 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 211"). ]
119 + CONTRACT=''
120 + getopts c: o
121 + CONTRACT=211
122 + getopts c: o
123 + [[ -n 211 ]]
124 + pgrep -c 211
125 + PARENT_PID=2281
126 + [[ -n 2281 ]]
127 + pgrep -P 2281
128 + CHILD_PID=2282
129 + [[ -n 2282 ]]
130 + kill -USR1 2282
1312026-03-24T17:53:43.558ZINFOslog-rs (mgd): handling smf refresh module = admin unit = smf
132 [ Mar 24 17:53:43 Method "refresh" exited with status 0. ]
1332026-03-24T17:53:43.561ZINFOslog-rs (mgd): starting stats server on smf refresh module = admin unit = smf
1342026-03-24T17:53:43.561ZINFOslog-rs (mgd): started stats server on smf refresh module = admin unit = smf
135 Mar 24 17:53:43.562 DEBG registered DTrace probes
136 Mar 24 17:53:43.562 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
137 Mar 24 17:53:43.562 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328
138 Mar 24 17:53:43.562 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
139 Mar 24 17:53:43.562 DEBG Nexus IP not provided, will use DNS to resolve it
140 Mar 24 17:53:43.562 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/internal-dns/resolver/src/resolver.rs:144
141 Mar 24 17:53:43.562 DEBG starting producer registration task
142 Mar 24 17:53:43.563 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: 431c0fa0-e69a-475f-bd42-810c754f4a4b, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:281
143 Mar 24 17:53:43.563 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
1442026-03-24T17:53:43.934ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::1]:65485 unit = api_server
1452026-03-24T17:53:43.934ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:65485 req_id = 53228978-f5f9-48ee-857c-be86a895a539 unit = api_server uri = /static/route4 version = 6.0.0
1462026-03-24T17:53:43.935ZINFOslog-rs (mgd): request completed latency_us = 589 local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:65485 req_id = 53228978-f5f9-48ee-857c-be86a895a539 response_code = 204 unit = api_server uri = /static/route4
1472026-03-24T17:53:43.935ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:65485 req_id = fcace5ff-c026-423f-8b30-367ad8c94119 unit = api_server uri = /static/route6 version = 6.0.0
1482026-03-24T17:53:43.935ZINFOslog-rs (mgd): request completed latency_us = 156 local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:65485 req_id = fcace5ff-c026-423f-8b30-367ad8c94119 response_code = 204 unit = api_server uri = /static/route6
1492026-03-24T17:53:43.936ZERROslog-rs (mg-lower): handling change failed: route lookup error io error No such process (os error 3) error = route lookup error io error No such process (os error 3) module = sync unit = event_loop
1502026-03-24T17:53:43.936ZINFOslog-rs (mg-lower): restarting sync loop module = sync unit = event_loop
151 Mar 24 17:53:48.565 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 675, tv_nsec: 192354607 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
152 Mar 24 17:53:48.566 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
153 Mar 24 17:53:48.566 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1542026-03-24T17:53:52.171ZDEBGslog-rs: client request body = None method = GET unit = gateway-client uri = http://[::1]:12225/local/switch-id
1552026-03-24T17:53:52.171ZDEBGslog-rs: client response result = Ok(Response { url: "http://[::1]:12225/local/switch-id", status: 200, headers: {"content-type": "application/json", "x-request-id": "0626bdf1-b9b5-4196-9abe-f1eef0c64f6e", "content-length": "26", "date": "Tue, 24 Mar 2026 17:53:52 GMT"} }) unit = gateway-client
1562026-03-24T17:53:52.171ZINFOslog-rs: we are in switch slot 0
157 Mar 24 17:54:03.396 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
158 Mar 24 17:54:03.396 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 20.004761132s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
159 Mar 24 17:54:03.597 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 690, tv_nsec: 395982503 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
160 Mar 24 17:54:03.598 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
161 Mar 24 17:54:03.598 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
162 Mar 24 17:54:18.639 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
163 Mar 24 17:54:18.639 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 35.207101465s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
164 Mar 24 17:54:19.314 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 706, tv_nsec: 70566809 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
165 Mar 24 17:54:19.315 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
166 Mar 24 17:54:19.315 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
167 Mar 24 17:54:34.357 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
168 Mar 24 17:54:34.357 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 50.882486792s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
169 Mar 24 17:54:35.548 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 722, tv_nsec: 259988325 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
170 Mar 24 17:54:35.548 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
171 Mar 24 17:54:35.548 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
172 Mar 24 17:54:50.591 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
173 Mar 24 17:54:50.591 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 67.07238707s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
174 Mar 24 17:54:52.169 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 738, tv_nsec: 836988442 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
175 Mar 24 17:54:52.170 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
176 Mar 24 17:54:52.170 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
177 Mar 24 17:55:07.212 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
178 Mar 24 17:55:07.212 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 83.648992947s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
179 Mar 24 17:55:10.892 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 757, tv_nsec: 509498434 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
180 Mar 24 17:55:10.893 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
181 Mar 24 17:55:10.893 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
182 Mar 24 17:55:25.935 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
183 Mar 24 17:55:25.935 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 102.321481838s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
184 Mar 24 17:55:36.793 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 783, tv_nsec: 339942951 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
185 Mar 24 17:55:36.793 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
186 Mar 24 17:55:36.793 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
187 Mar 24 17:55:51.835 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
188 Mar 24 17:55:51.835 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 128.151766961s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
189 Mar 24 17:56:13.864 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 820, tv_nsec: 311559837 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
190 Mar 24 17:56:13.865 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
191 Mar 24 17:56:13.865 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
192 Mar 24 17:56:28.906 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
193 Mar 24 17:56:28.906 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 165.123269424s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
194 Mar 24 17:57:11.546 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 877, tv_nsec: 898429658 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
195 Mar 24 17:57:11.547 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
196 Mar 24 17:57:11.547 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
197 Mar 24 17:57:26.552 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
198 Mar 24 17:57:26.554 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 222.71581226s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
199 Mar 24 17:58:16.177 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("10d64789-f5ab-434b-8575-aa99adfc92f7.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c22557a-edad-4146-92a1-18f621b76499.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("9e84d014-d38f-4477-abee-8f1b237f9801.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 942, tv_nsec: 525182319 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
200 Mar 24 17:58:16.197 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
201 Mar 24 17:58:16.209 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
202 Mar 24 17:58:31.210 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
203 Mar 24 17:58:31.213 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 287.375072172s, count: 9, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
2042026-03-24T17:58:57.485ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::a]:39179 unit = api_server
2052026-03-24T17:58:57.494ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::c]:43959 unit = api_server
2062026-03-24T17:58:57.501ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:43959 req_id = a2f52427-27de-429e-8e38-15911b22005d unit = api_server uri = /bfd/peers version = 6.0.0
2072026-03-24T17:58:57.501ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:39179 req_id = c6851948-2b0f-4086-a925-e5fb7cb85c4d unit = api_server uri = /bfd/peers version = 6.0.0
2082026-03-24T17:58:57.512ZINFOslog-rs (mgd): request completed latency_us = 14842 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:39179 req_id = c6851948-2b0f-4086-a925-e5fb7cb85c4d response_code = 200 unit = api_server uri = /bfd/peers
2092026-03-24T17:58:57.512ZINFOslog-rs (mgd): request completed latency_us = 14852 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:43959 req_id = a2f52427-27de-429e-8e38-15911b22005d response_code = 200 unit = api_server uri = /bfd/peers
2102026-03-24T17:58:57.846ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::b]:63480 unit = api_server
2112026-03-24T17:58:57.846ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:63480 req_id = d9e0e637-4cf5-42b3-8d2e-108f0e6d9be5 unit = api_server uri = /bfd/peers version = 6.0.0
2122026-03-24T17:58:57.846ZINFOslog-rs (mgd): request completed latency_us = 47 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:63480 req_id = d9e0e637-4cf5-42b3-8d2e-108f0e6d9be5 response_code = 200 unit = api_server uri = /bfd/peers
2132026-03-24T17:59:27.415ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::a]:54494 unit = api_server
2142026-03-24T17:59:27.416ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:54494 req_id = 776bd584-fd88-43ce-8b6b-fa351f85c6e2 unit = api_server uri = /bfd/peers version = 6.0.0
2152026-03-24T17:59:27.416ZINFOslog-rs (mgd): request completed latency_us = 74 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:54494 req_id = 776bd584-fd88-43ce-8b6b-fa351f85c6e2 response_code = 200 unit = api_server uri = /bfd/peers
2162026-03-24T17:59:27.417ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::c]:54589 unit = api_server
2172026-03-24T17:59:27.418ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:54589 req_id = a072a47a-cc51-4583-a090-f5ef10a7db97 unit = api_server uri = /bfd/peers version = 6.0.0
2182026-03-24T17:59:27.418ZINFOslog-rs (mgd): request completed latency_us = 63 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:54589 req_id = a072a47a-cc51-4583-a090-f5ef10a7db97 response_code = 200 unit = api_server uri = /bfd/peers
2192026-03-24T17:59:27.631ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::a]:56446 unit = api_server
2202026-03-24T17:59:27.631ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:56446 req_id = b5a52dfc-6d77-4cd9-86ad-175b7ef8551a unit = api_server uri = /static/route4 version = 6.0.0
2212026-03-24T17:59:27.637ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::c]:40631 unit = api_server
2222026-03-24T17:59:27.637ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:40631 req_id = a72ea786-8b97-47ce-b750-4e892af89839 unit = api_server uri = /static/route4 version = 6.0.0
2232026-03-24T17:59:27.642ZINFOslog-rs (mgd): request completed latency_us = 5148 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:40631 req_id = a72ea786-8b97-47ce-b750-4e892af89839 response_code = 200 unit = api_server uri = /static/route4
2242026-03-24T17:59:27.642ZINFOslog-rs (mgd): request completed latency_us = 11510 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:56446 req_id = b5a52dfc-6d77-4cd9-86ad-175b7ef8551a response_code = 200 unit = api_server uri = /static/route4
2252026-03-24T17:59:27.643ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:40631 req_id = 2fb8a0df-bdaf-46ba-9caa-d05375f63c99 unit = api_server uri = /static/route6 version = 6.0.0
2262026-03-24T17:59:27.643ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:56446 req_id = 72562657-fd90-41af-af7c-a3c5f1b67b1b unit = api_server uri = /static/route6 version = 6.0.0
2272026-03-24T17:59:27.643ZINFOslog-rs (mgd): request completed latency_us = 247 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:40631 req_id = 2fb8a0df-bdaf-46ba-9caa-d05375f63c99 response_code = 200 unit = api_server uri = /static/route6
2282026-03-24T17:59:27.643ZINFOslog-rs (mgd): request completed latency_us = 44 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:56446 req_id = 72562657-fd90-41af-af7c-a3c5f1b67b1b response_code = 200 unit = api_server uri = /static/route6
2292026-03-24T17:59:27.734ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::b]:40994 unit = api_server
2302026-03-24T17:59:27.734ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:40994 req_id = 4540fbcc-733a-4885-ae63-071d43b062f5 unit = api_server uri = /bfd/peers version = 6.0.0
2312026-03-24T17:59:27.734ZINFOslog-rs (mgd): request completed latency_us = 55 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:40994 req_id = 4540fbcc-733a-4885-ae63-071d43b062f5 response_code = 200 unit = api_server uri = /bfd/peers
2322026-03-24T17:59:28.014ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::b]:45555 unit = api_server
2332026-03-24T17:59:28.014ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:45555 req_id = 0f63373c-21a8-4f05-84aa-71592109bbdb unit = api_server uri = /static/route4 version = 6.0.0
2342026-03-24T17:59:28.014ZINFOslog-rs (mgd): request completed latency_us = 82 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:45555 req_id = 0f63373c-21a8-4f05-84aa-71592109bbdb response_code = 200 unit = api_server uri = /static/route4
2352026-03-24T17:59:28.014ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:45555 req_id = 3444cd48-2920-4046-a412-5074638d52cb unit = api_server uri = /static/route6 version = 6.0.0
2362026-03-24T17:59:28.015ZINFOslog-rs (mgd): request completed latency_us = 74 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:45555 req_id = 3444cd48-2920-4046-a412-5074638d52cb response_code = 200 unit = api_server uri = /static/route6
2372026-03-24T17:59:57.408ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::a]:32868 unit = api_server
2382026-03-24T17:59:57.408ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:32868 req_id = c64cee91-8628-418f-9ad5-7c0f1fcb74ae unit = api_server uri = /bfd/peers version = 6.0.0
2392026-03-24T17:59:57.408ZINFOslog-rs (mgd): request completed latency_us = 41 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:32868 req_id = c64cee91-8628-418f-9ad5-7c0f1fcb74ae response_code = 200 unit = api_server uri = /bfd/peers
2402026-03-24T17:59:57.413ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::c]:45627 unit = api_server
2412026-03-24T17:59:57.413ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:45627 req_id = e76af731-a168-4049-a6e9-e5447e5bf6d7 unit = api_server uri = /bfd/peers version = 6.0.0
2422026-03-24T17:59:57.413ZINFOslog-rs (mgd): request completed latency_us = 39 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:45627 req_id = e76af731-a168-4049-a6e9-e5447e5bf6d7 response_code = 200 unit = api_server uri = /bfd/peers
2432026-03-24T17:59:57.572ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::c]:35220 unit = api_server
2442026-03-24T17:59:57.572ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:35220 req_id = 5c557168-c60e-49a1-9e3d-8b1122a43d54 unit = api_server uri = /static/route4 version = 6.0.0
2452026-03-24T17:59:57.572ZINFOslog-rs (mgd): request completed latency_us = 103 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:35220 req_id = 5c557168-c60e-49a1-9e3d-8b1122a43d54 response_code = 200 unit = api_server uri = /static/route4
2462026-03-24T17:59:57.572ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:35220 req_id = 2b6d0015-eaaf-4389-b5f5-35633a3414c2 unit = api_server uri = /static/route6 version = 6.0.0
2472026-03-24T17:59:57.572ZINFOslog-rs (mgd): request completed latency_us = 44 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::c]:35220 req_id = 2b6d0015-eaaf-4389-b5f5-35633a3414c2 response_code = 200 unit = api_server uri = /static/route6
2482026-03-24T17:59:57.647ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::a]:34062 unit = api_server
2492026-03-24T17:59:57.648ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:34062 req_id = 55159874-689b-4542-9159-b3a3a30641f7 unit = api_server uri = /static/route4 version = 6.0.0
2502026-03-24T17:59:57.648ZINFOslog-rs (mgd): request completed latency_us = 82 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:34062 req_id = 55159874-689b-4542-9159-b3a3a30641f7 response_code = 200 unit = api_server uri = /static/route4
2512026-03-24T17:59:57.648ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:34062 req_id = 9ff720fc-f4a7-47bd-b512-ea9b0f322f85 unit = api_server uri = /static/route6 version = 6.0.0
2522026-03-24T17:59:57.648ZINFOslog-rs (mgd): request completed latency_us = 61 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::a]:34062 req_id = 9ff720fc-f4a7-47bd-b512-ea9b0f322f85 response_code = 200 unit = api_server uri = /static/route6
2532026-03-24T17:59:57.735ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::b]:36234 unit = api_server
2542026-03-24T17:59:57.735ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:36234 req_id = ddc6d681-17c8-43a6-968b-6bb70d5f76d1 unit = api_server uri = /bfd/peers version = 6.0.0
2552026-03-24T17:59:57.735ZINFOslog-rs (mgd): request completed latency_us = 65 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:36234 req_id = ddc6d681-17c8-43a6-968b-6bb70d5f76d1 response_code = 200 unit = api_server uri = /bfd/peers
2562026-03-24T17:59:57.951ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::b]:57818 unit = api_server
2572026-03-24T17:59:57.951ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:57818 req_id = 2b5efe7e-7b97-4c31-abe5-cffee70ca53a unit = api_server uri = /static/route4 version = 6.0.0
2582026-03-24T17:59:57.951ZINFOslog-rs (mgd): request completed latency_us = 72 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:57818 req_id = 2b5efe7e-7b97-4c31-abe5-cffee70ca53a response_code = 200 unit = api_server uri = /static/route4
2592026-03-24T17:59:57.951ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:57818 req_id = ede1b22d-8b24-495b-b48a-67e4109efbcd unit = api_server uri = /static/route6 version = 6.0.0
2602026-03-24T17:59:57.951ZINFOslog-rs (mgd): request completed latency_us = 37 local_addr = [::]:4676 method = GET module = admin remote_addr = [fd00:1122:3344:101::b]:57818 req_id = ede1b22d-8b24-495b-b48a-67e4109efbcd response_code = 200 unit = api_server uri = /static/route6