Max level shown:
1 [ Jun 9 02:16:05 Disabled. ]
2 [ Jun 9 02:16:05 Rereading configuration. ]
3 [ Jun 9 02:16:05 Enabled. ]
4 [ Jun 9 02:16:05 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
5 [ Jun 9 02:16:05 Method "start" exited with status 0. ]
62026-06-09T02:16:05.109ZINFOslog-rs: signal handler waiting for context
72026-06-09T02:16:05.126ZINFOslog-rs: signal handler waiting got context
82026-06-09T02:16:05.126ZINFOslog-rs: [net1] sm initialized with addr fe80::6852:29ff:fe69:e691 on if net1 index 4
92026-06-09T02:16:05.126ZINFOslog-rs: [net0] sm initialized with addr fe80::ac60:cff:fe6f:238e on if net0 index 3
102026-06-09T02:16:05.126ZDEBGslog-rs: [net1] starting discovery handler
112026-06-09T02:16:05.126ZDEBGslog-rs: [net0] starting discovery handler
122026-06-09T02:16:05.126ZINFOslog-rs: admin: listening on [::1]:8000
132026-06-09T02:16:05.126ZINFOslog-rs: admin: server started
142026-06-09T02:16:05.573ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }}))
152026-06-09T02:16:05.573ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }}))
162026-06-09T02:16:06.664ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }}))
172026-06-09T02:16:06.664ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }}))
182026-06-09T02:16:22.112ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }}))
192026-06-09T02:16:22.112ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }}))
202026-06-09T02:16:22.113ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }, Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }}))
212026-06-09T02:16:22.113ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }, Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }}))
222026-06-09T02:16:29.127ZINFOslog-rs: [net0] nbr is fe80::2cee:99ff:fea4:3496@oxz_switch transit
232026-06-09T02:16:29.127ZDEBGslog-rs: [net0] transition solicit -> exchange
242026-06-09T02:16:29.127ZINFOslog-rs: [net0] exchange: listening on [fe80::ac60:cff:fe6f:238e]:56797
252026-06-09T02:16:29.127ZINFOslog-rs: [net0] waiting for exchange server to start
262026-06-09T02:16:30.406ZINFOslog-rs: [net0] sending 1 routes to illumos
272026-06-09T02:16:30.406ZINFOslog-rs: [net0] adding route fdb3:5011:df19:aedd::1 -> fe80::2cee:99ff:fea4:3496
282026-06-09T02:16:30.406ZINFOslog-rs: [net0] removing 0 routes from illumos
292026-06-09T02:16:30.407ZINFOslog-rs: [net0] adding tunnel route 0.0.0.0/0 -[99]-> fdb3:5011:df19:aedd::1
302026-06-09T02:16:47.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
312026-06-09T02:16:49.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
322026-06-09T02:16:51.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
332026-06-09T02:16:53.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
342026-06-09T02:16:55.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
352026-06-09T02:16:57.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
362026-06-09T02:16:59.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
372026-06-09T02:17:01.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
382026-06-09T02:17:03.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
392026-06-09T02:17:05.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
40 Jun 09 02:17:06.325 DEBG registered DTrace probes
41 Jun 09 02:17:06.326 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot
42 Jun 09 02:17:06.326 INFO listening, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
43 Jun 09 02:17:06.326 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot
44 Jun 09 02:17:06.326 DEBG Nexus IP not provided, will use DNS to resolve it
45 Jun 09 02:17:06.326 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
46 Jun 09 02:17:06.326 DEBG starting producer registration task
47 Jun 09 02:17:06.326 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::1]:8001, producer_id: ea7d06fa-114e-485e-abec-9f738f0a68f7, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
48 Jun 09 02:17:06.326 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
49 Jun 09 02:17:06.327 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 746, tv_nsec: 816369748 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
50 Jun 09 02:17:06.328 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
51 Jun 09 02:17:06.329 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
522026-06-09T02:17:07.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
532026-06-09T02:17:09.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
542026-06-09T02:17:11.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
552026-06-09T02:17:13.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
562026-06-09T02:17:15.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
572026-06-09T02:17:17.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
582026-06-09T02:17:19.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
59 Jun 09 02:17:21.289 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
60 Jun 09 02:17:21.290 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 15.004170664s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
61 Jun 09 02:17:21.428 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 761, tv_nsec: 958773949 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
62 Jun 09 02:17:21.429 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
63 Jun 09 02:17:21.430 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
642026-06-09T02:17:21.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
652026-06-09T02:17:23.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
662026-06-09T02:17:25.935ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
672026-06-09T02:17:27.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
682026-06-09T02:17:29.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
692026-06-09T02:17:31.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
702026-06-09T02:17:33.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
712026-06-09T02:17:35.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
72 Jun 09 02:17:36.391 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
73 Jun 09 02:17:36.391 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 30.146368228s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
74 Jun 09 02:17:36.952 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 777, tv_nsec: 524885020 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
75 Jun 09 02:17:36.953 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
76 Jun 09 02:17:36.954 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
772026-06-09T02:17:37.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
782026-06-09T02:17:39.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
792026-06-09T02:17:41.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
802026-06-09T02:17:43.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
812026-06-09T02:17:45.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
822026-06-09T02:17:47.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
832026-06-09T02:17:49.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
84 Jun 09 02:17:51.914 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
85 Jun 09 02:17:51.914 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 45.711807104s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
862026-06-09T02:17:51.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
87 Jun 09 02:17:52.772 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 793, tv_nsec: 387435345 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
88 Jun 09 02:17:52.773 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
89 Jun 09 02:17:52.773 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
902026-06-09T02:17:53.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
912026-06-09T02:17:55.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
922026-06-09T02:17:57.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
932026-06-09T02:17:59.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
942026-06-09T02:18:01.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
952026-06-09T02:18:03.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
962026-06-09T02:18:05.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
97 Jun 09 02:18:07.734 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
98 Jun 09 02:18:07.734 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 61.574687147s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
992026-06-09T02:18:07.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
100 Jun 09 02:18:08.999 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 809, tv_nsec: 658903506 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
101 Jun 09 02:18:09.000 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
102 Jun 09 02:18:09.001 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1032026-06-09T02:18:09.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1042026-06-09T02:18:11.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1052026-06-09T02:18:13.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1062026-06-09T02:18:15.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1072026-06-09T02:18:17.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1082026-06-09T02:18:19.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1092026-06-09T02:18:21.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1102026-06-09T02:18:23.936ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
111 Jun 09 02:18:23.961 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
112 Jun 09 02:18:23.961 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 77.845990675s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1132026-06-09T02:18:25.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1142026-06-09T02:18:27.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
115 Jun 09 02:18:28.044 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 828, tv_nsec: 755578561 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
116 Jun 09 02:18:28.045 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
117 Jun 09 02:18:28.046 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1182026-06-09T02:18:29.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1192026-06-09T02:18:31.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1202026-06-09T02:18:33.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1212026-06-09T02:18:35.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1222026-06-09T02:18:37.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1232026-06-09T02:18:39.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1242026-06-09T02:18:41.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
125 Jun 09 02:18:43.006 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
126 Jun 09 02:18:43.006 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 96.942060465s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1272026-06-09T02:18:43.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
1282026-06-09T02:18:45.937ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
129 Jun 09 02:18:47.570 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 848, tv_nsec: 334343188 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
130 Jun 09 02:18:47.571 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
131 Jun 09 02:18:47.572 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
132 Jun 09 02:19:02.533 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
133 Jun 09 02:19:02.533 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 116.522156554s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1342026-06-09T02:19:13.980ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
135 Jun 09 02:19:15.787 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 876, tv_nsec: 619358922 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
136 Jun 09 02:19:15.788 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
137 Jun 09 02:19:15.789 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
138 Jun 09 02:19:30.792 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
139 Jun 09 02:19:30.792 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 144.808169589s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
140 Jun 09 02:20:15.381 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 936, tv_nsec: 211298153 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
141 Jun 09 02:20:15.387 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
142 Jun 09 02:20:15.422 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1432026-06-09T02:20:21.701ZWARNslog-rs: [net0] neighbor oxz_switch@fe80::2cee:99ff:fea4:3496 missed solicit interval
144 Jun 09 02:20:30.520 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
145 Jun 09 02:20:30.558 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 204.573506853s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
146 Jun 09 02:21:14.770 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1bf57f6b-e73a-4338-a384-5d2c11ddb2c1.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5a7052cd-ea15-4706-ad66-604c31f239a9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("79b3cedd-cd80-4b2c-9704-89b7545bbb9d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 995, tv_nsec: 600695231 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
147 Jun 09 02:21:14.772 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
148 Jun 09 02:21:14.777 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
149 Jun 09 02:21:15.011 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "02d0b600-152a-4fb0-b91a-9fceff4be31b", "content-length": "41", "date": "Tue, 09 Jun 2026 02:21:15 GMT"} }), component: producer-registration-task
150 Jun 09 02:21:15.011 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
151 Jun 09 02:21:26.815 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057
152 Jun 09 02:21:26.819 INFO request completed, latency_us: 3268, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: ffcb5d7c-67a2-4806-a520-8c90959e237c, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
153 Jun 09 02:21:27.815 INFO request completed, latency_us: 987, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 3d152c2b-c0eb-4f38-9356-f46d93e40e27, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
154 Jun 09 02:21:28.816 INFO request completed, latency_us: 1200, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 989133be-133c-4c51-9080-280c4fae0134, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
155 Jun 09 02:21:29.816 INFO request completed, latency_us: 1040, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: dd1066ff-b104-4705-8557-ddaa3edb68c4, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
156 Jun 09 02:21:30.816 INFO request completed, latency_us: 1009, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: a7679e39-9060-4eba-88f1-6bb56f7af2b7, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
157 Jun 09 02:21:31.816 INFO request completed, latency_us: 949, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 09f0e95b-bf61-46ea-b772-921d881fbe0a, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
158 Jun 09 02:21:32.816 INFO request completed, latency_us: 1003, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 845fa2ba-0777-4988-819f-a3b4f3077b5a, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
159 Jun 09 02:21:33.816 INFO request completed, latency_us: 1018, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: cb8893f7-10e7-43a9-972e-1dee08759b3e, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
160 Jun 09 02:21:34.815 INFO request completed, latency_us: 1052, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 758a1d59-6909-4b46-9890-3c45e05dfdb3, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
161 Jun 09 02:21:35.816 INFO request completed, latency_us: 1185, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 3e4a4e54-0adf-46a3-9427-245684bc8356, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
162 Jun 09 02:21:36.816 INFO request completed, latency_us: 1003, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 5d7f12db-d57d-4065-93b1-0dc7161a687e, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
163 Jun 09 02:21:38.022 INFO request completed, latency_us: 1344, response_code: 200, uri: /ea7d06fa-114e-485e-abec-9f738f0a68f7, method: GET, req_id: 9882919a-a121-4e93-9538-9652c0363761, remote_addr: [fd00:1122:3344:101::d]:49131, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874