Max level shown:
1 [ Jan 5 23:15:53 Disabled. ]
2 [ Jan 5 23:15:53 Rereading configuration. ]
3 [ Jan 5 23:15:54 Enabled. ]
4 [ Jan 5 23:15:54 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
5 [ Jan 5 23:15:54 Method "start" exited with status 0. ]
6 [ Jan 5 23:15:54 Rereading configuration. ]
7 [ Jan 5 23:15:54 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 215"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=215
11 + getopts c: o
12 + [[ -n 215 ]]
13 + pgrep -c 215
14 + PARENT_PID=2288
15 + [[ -n 2288 ]]
16 + pgrep -P 2288
17 + CHILD_PID=2289
18 + [[ -n 2289 ]]
19 + kill -USR1 2289
20 [ Jan 5 23:15:54 Method "refresh" exited with status 0. ]
21 [ Jan 5 23:15:54 Stopping because all processes in service exited. ]
22 [ Jan 5 23:15:54 Executing stop method (:kill). ]
23 [ Jan 5 23:15:55 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mgd/pkg/mgd_method_script.sh &"). ]
24 [ Jan 5 23:15:55 Method "start" exited with status 0. ]
252026-01-05T23:15:55.056ZINFOslog-rs (mgd): signal handler waiting for context module = admin unit = signal
262026-01-05T23:15:55.057ZINFOslog-rs (bgp): dispatcher started listen_address = [::]:179 module = neighbor unit = dispatcher
272026-01-05T23:15:55.057ZDEBGslog-rs (bgp): listener bind: [::]:179 listen_address = [::]:179 module = neighbor unit = dispatcher
282026-01-05T23:15:55.076ZINFOslog-rs (mgd): signal handler got context module = admin unit = signal
292026-01-05T23:15:55.076ZINFOslog-rs (mgd): starting bgp routers: {} module = admin unit = daemon
302026-01-05T23:15:55.076ZINFOslog-rs (mgd): starting bfd sessions: [] module = admin unit = daemon
312026-01-05T23:15:55.077ZERROslog-rs (mgd): failed to start stats server: underlay address not found error = underlay address not found module = admin params = hostname oxz_switch, rack 9c7eed32-7b0a-49c1-8d19-76f35fc7d47f, sled ef3426e3-4fa9-4d16-b2c2-c5b82b132208 unit = daemon
322026-01-05T23:15:55.078ZINFOslog-rs: listening on [::]:4676
332026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
342026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bestpath/config/fanout unit = api_server versions = all ending with 2.0.0
352026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bfd/peers unit = api_server versions = all
362026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bfd/peers unit = api_server versions = all
372026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bfd/peers/{addr} unit = api_server versions = all
382026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/clear/neighbor unit = api_server versions = all
392026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/checker unit = api_server versions = all
402026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/checker unit = api_server versions = all
412026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/checker unit = api_server versions = all
422026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/checker unit = api_server versions = all
432026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/neighbor unit = api_server versions = all
442026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbor unit = api_server versions = all
452026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/neighbor unit = api_server versions = all
462026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/neighbor unit = api_server versions = all
472026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/neighbors unit = api_server versions = all
482026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin4 unit = api_server versions = all
492026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin4 unit = api_server versions = all
502026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin4 unit = api_server versions = all
512026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin4 unit = api_server versions = all
522026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
532026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
542026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
552026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/origin6 unit = api_server versions = all starting from 2.0.0
562026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/router unit = api_server versions = all
572026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/router unit = api_server versions = all
582026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/router unit = api_server versions = all
592026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/router unit = api_server versions = all
602026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/routers unit = api_server versions = all
612026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /bgp/config/shaper unit = api_server versions = all
622026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/config/shaper unit = api_server versions = all
632026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/config/shaper unit = api_server versions = all
642026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /bgp/config/shaper unit = api_server versions = all
652026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/fsm unit = api_server versions = all starting from 2.0.0
662026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/history/message unit = api_server versions = all starting from 2.0.0
672026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/message-history unit = api_server versions = all ending with 2.0.0
682026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /bgp/omicron/apply unit = api_server versions = all
692026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/exported unit = api_server versions = all
702026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/imported unit = api_server versions = all ending with 2.0.0
712026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all ending with 2.0.0
722026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/neighbors unit = api_server versions = all starting from 2.0.0
732026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /bgp/status/selected unit = api_server versions = all ending with 2.0.0
742026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
752026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = POST module = admin path = /rib/config/bestpath/fanout unit = api_server versions = all starting from 2.0.0
762026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/imported unit = api_server versions = all starting from 2.0.0
772026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /rib/status/selected unit = api_server versions = all starting from 2.0.0
782026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route4 unit = api_server versions = all
792026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route4 unit = api_server versions = all
802026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route4 unit = api_server versions = all
812026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = DELETE module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
822026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = GET module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
832026-01-05T23:15:55.079ZDEBGslog-rs (mgd): registered endpoint local_addr = [::]:4676 method = PUT module = admin path = /static/route6 unit = api_server versions = all starting from 2.0.0
842026-01-05T23:15:55.079ZINFOslog-rs (mgd): listening local_addr = [::]:4676 module = admin unit = api_server
852026-01-05T23:15:55.079ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
862026-01-05T23:15:55.079ZDEBGslog-rs (mgd): successfully registered DTrace USDT probes local_addr = [::]:4676 module = admin unit = api_server
872026-01-05T23:15:55.079ZINFOslog-rs: admin: server started
882026-01-05T23:15:56.080ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
892026-01-05T23:15:57.080ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
902026-01-05T23:15:58.080ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
912026-01-05T23:15:59.081ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
922026-01-05T23:16:00.081ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
932026-01-05T23:16:01.219ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
942026-01-05T23:16:02.220ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
952026-01-05T23:16:03.220ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
962026-01-05T23:16:04.220ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
972026-01-05T23:16:05.221ZWARNslog-rs (mg-lower): failed to ensure TEP address fde6:3bab:3678:69e3::1 on ASIC: Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) error = Communication Error: error sending request for url (http://localhost:12224/loopback/ipv6) module = sync prefix = fde6:3bab:3678:69e3::1 unit = dpd
98 [ Jan 5 23:16:06 Rereading configuration. ]
99 [ Jan 5 23:16:06 Executing refresh method ("/opt/oxide/mgd/pkg/refresh_ctrun_child.sh -c 220"). ]
100 + CONTRACT=''
101 + getopts c: o
102 + CONTRACT=220
103 + getopts c: o
104 + [[ -n 220 ]]
105 + pgrep -c 220
106 + PARENT_PID=2316
107 + [[ -n 2316 ]]
108 + pgrep -P 2316
109 + CHILD_PID=2317
110 + [[ -n 2317 ]]
111 + kill -USR1 2317
1122026-01-05T23:16:06.379ZINFOslog-rs (mgd): handling smf refresh module = admin unit = smf
113 [ Jan 5 23:16:06 Method "refresh" exited with status 0. ]
1142026-01-05T23:16:06.381ZINFOslog-rs (mgd): starting stats server on smf refresh module = admin unit = smf
1152026-01-05T23:16:06.381ZINFOslog-rs (mgd): started stats server on smf refresh module = admin unit = smf
116 Jan 05 23:16:06.382 DEBG registered DTrace probes
117 Jan 05 23:16:06.382 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
118 Jan 05 23:16:06.382 INFO listening, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.5/src/server.rs:328
119 Jan 05 23:16:06.383 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:4677, component: dropshot
120 Jan 05 23:16:06.383 DEBG Nexus IP not provided, will use DNS to resolve it
121 Jan 05 23:16:06.383 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/internal-dns/resolver/src/resolver.rs:144
122 Jan 05 23:16:06.383 DEBG starting producer registration task
123 Jan 05 23:16:06.383 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:4677, producer_id: fa1e0b42-be1e-4f43-96ec-f8cca9a4c140, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:279
124 Jan 05 23:16:06.383 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
1252026-01-05T23:16:06.764ZINFOslog-rs (mgd): accepted connection local_addr = [::]:4676 module = admin remote_addr = [fd00:1122:3344:101::1]:55654 unit = api_server
1262026-01-05T23:16:06.764ZDEBGslog-rs (mgd): determined request API version local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:55654 req_id = a05ef950-6352-4ebf-b111-e76c86cca6ae unit = api_server uri = /static/route4 version = 2.0.0
1272026-01-05T23:16:06.765ZINFOslog-rs (mgd): request completed latency_us = 547 local_addr = [::]:4676 method = PUT module = admin remote_addr = [fd00:1122:3344:101::1]:55654 req_id = a05ef950-6352-4ebf-b111-e76c86cca6ae response_code = 204 unit = api_server uri = /static/route4
1282026-01-05T23:16:06.766ZERROslog-rs (mg-lower): handling change failed: libnet error route io error No such process (os error 3) error = libnet error route io error No such process (os error 3) module = sync unit = event_loop
1292026-01-05T23:16:06.766ZINFOslog-rs (mg-lower): restarting sync loop module = sync unit = event_loop
130 Jan 05 23:16:11.386 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 655, tv_nsec: 24938724 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
131 Jan 05 23:16:11.386 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
132 Jan 05 23:16:11.386 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
133 Jan 05 23:16:26.388 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
134 Jan 05 23:16:26.388 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 20.004509661s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
135 Jan 05 23:16:26.693 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 670, tv_nsec: 332114760 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
136 Jan 05 23:16:26.693 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
137 Jan 05 23:16:26.694 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
138 Jan 05 23:16:41.694 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
139 Jan 05 23:16:41.694 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 35.311285909s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
140 Jan 05 23:16:42.088 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 685, tv_nsec: 726996933 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
141 Jan 05 23:16:42.088 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
142 Jan 05 23:16:42.088 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
143 Jan 05 23:16:57.088 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
144 Jan 05 23:16:57.088 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 50.705254363s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
145 Jan 05 23:16:57.998 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 701, tv_nsec: 636984737 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
146 Jan 05 23:16:57.998 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
147 Jan 05 23:16:57.998 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
148 Jan 05 23:17:13.000 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
149 Jan 05 23:17:13.000 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 66.616693916s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
150 Jan 05 23:17:15.970 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 719, tv_nsec: 609251922 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
151 Jan 05 23:17:15.970 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
152 Jan 05 23:17:15.971 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
153 Jan 05 23:17:30.972 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
154 Jan 05 23:17:30.972 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 84.589198546s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
155 Jan 05 23:17:36.793 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 740, tv_nsec: 432700797 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
156 Jan 05 23:17:36.794 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
157 Jan 05 23:17:36.794 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
158 Jan 05 23:17:51.795 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
159 Jan 05 23:17:51.795 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 105.411930727s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
160 Jan 05 23:18:01.108 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 764, tv_nsec: 747401001 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
161 Jan 05 23:18:01.109 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
162 Jan 05 23:18:01.109 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
163 Jan 05 23:18:16.110 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
164 Jan 05 23:18:16.110 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 129.726764584s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
165 Jan 05 23:18:36.879 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 800, tv_nsec: 518309492 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
166 Jan 05 23:18:36.879 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
167 Jan 05 23:18:36.879 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
168 Jan 05 23:18:51.881 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
169 Jan 05 23:18:51.881 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 165.497762727s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
170 Jan 05 23:19:36.751 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 860, tv_nsec: 390024597 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
171 Jan 05 23:19:36.751 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
172 Jan 05 23:19:36.751 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
173 Jan 05 23:19:51.753 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
174 Jan 05 23:19:51.753 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 225.369753477s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371
175 Jan 05 23:20:33.025 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("081fc1bb-8fa2-487d-b403-50f00ff69504.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("43fa1956-f739-482a-aa50-9e4df28e05f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("86a5d9ac-2ffd-4db6-a67b-bd409b3df5fd.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 916, tv_nsec: 664313002 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
176 Jan 05 23:20:33.025 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
177 Jan 05 23:20:33.026 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
178 Jan 05 23:20:48.027 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
179 Jan 05 23:20:48.027 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)", delay: 281.644129034s, count: 9, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/b36f032/oximeter/producer/src/lib.rs:371