Max level shown:
1 [ May 20 17:40:57 Disabled. ]
2 [ May 20 17:40:57 Rereading configuration. ]
3 [ May 20 17:40:58 Enabled. ]
4 [ May 20 17:40:58 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
5 [ May 20 17:40:58 Method "start" exited with status 0. ]
6 [ May 20 17:40:58 Rereading configuration. ]
7 [ May 20 17:40:58 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 212"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=212
11 + getopts c: o
12 + [[ -n 212 ]]
13 + pgrep -c 212
14 + PARENT_PID=2285
15 + [[ -n 2285 ]]
16 + pgrep -P 2285
17 + CHILD_PID=2286
18 + [[ -n 2286 ]]
19 + kill -USR1 2286
20 [ May 20 17:40:58 Method "refresh" exited with status 0. ]
21 [ May 20 17:40:58 Stopping because all processes in service exited. ]
22 [ May 20 17:40:58 Executing stop method (:kill). ]
23 [ May 20 17:40:58 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
24 [ May 20 17:40:58 Method "start" exited with status 0. ]
252026-05-20T17:40:58.264ZINFOslog-rs: signal handler waiting for context
262026-05-20T17:40:58.285ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
272026-05-20T17:40:58.285ZERROslog-rs: failed to start stats server: underlay address not found
282026-05-20T17:40:58.285ZINFOslog-rs: signal handler waiting got context
292026-05-20T17:40:58.286ZINFOslog-rs: admin: listening on [::1]:8000
302026-05-20T17:40:58.286ZINFOslog-rs: admin: server started
312026-05-20T17:40:59.285ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
322026-05-20T17:41:00.285ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
332026-05-20T17:41:01.285ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
342026-05-20T17:41:02.285ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
352026-05-20T17:41:03.285ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
362026-05-20T17:41:04.291ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
372026-05-20T17:41:05.291ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
382026-05-20T17:41:06.291ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
392026-05-20T17:41:07.291ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
402026-05-20T17:41:08.292ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
412026-05-20T17:41:09.292ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::a01c:41ff:fee1:90c8 on if tfportrear0_0 index 3
422026-05-20T17:41:09.292ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
432026-05-20T17:41:09.292ZINFOslog-rs: [tfportrear0_0] nbr is fe80::f00c:acff:fefb:6fee@buskin server
442026-05-20T17:41:09.292ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
452026-05-20T17:41:09.292ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::a01c:41ff:fee1:90c8]:56797
462026-05-20T17:41:09.293ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
47 [ May 20 17:41:09 Rereading configuration. ]
48 [ May 20 17:41:09 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 217"). ]
49 + CONTRACT=''
50 + getopts c: o
51 + CONTRACT=217
52 + getopts c: o
53 + [[ -n 217 ]]
54 + pgrep -c 217
55 + PARENT_PID=2308
56 + [[ -n 2308 ]]
57 + pgrep -P 2308
58 + CHILD_PID=2309
59 + [[ -n 2309 ]]
60 + kill -USR1 2309
612026-05-20T17:41:09.767ZINFOslog-rs: handling smf refresh
62 [ May 20 17:41:09 Method "refresh" exited with status 0. ]
632026-05-20T17:41:09.771ZINFOslog-rs: starting stats server on smf refresh
64 May 20 17:41:09.772 DEBG registered DTrace probes
65 May 20 17:41:09.772 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
66 May 20 17:41:09.772 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328
67 May 20 17:41:09.772 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
68 May 20 17:41:09.773 DEBG Nexus IP not provided, will use DNS to resolve it
69 May 20 17:41:09.773 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/internal-dns/resolver/src/resolver.rs:144
70 May 20 17:41:09.773 DEBG starting producer registration task
71 May 20 17:41:09.773 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: ef5384d3-e66d-454a-b152-dbe827806e72, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:283
72 May 20 17:41:09.773 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
732026-05-20T17:41:10.976ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd1a:54b9:a8ba:8a26::1, vni: 99, metric: 18446744073709551615, }, }
742026-05-20T17:41:11.294ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
752026-05-20T17:41:11.295ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
762026-05-20T17:41:11.295ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
772026-05-20T17:41:11.296ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::f00c:acff:fefb:6fee on port Rear(Rear("rear0"))/0
782026-05-20T17:41:11.297ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::f00c:acff:fefb:6fee on port Rear(Rear("rear0"))/0
792026-05-20T17:41:11.298ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::f00c:acff:fefb:6fee on port Rear(Rear("rear0"))/0
802026-05-20T17:41:11.298ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::f00c:acff:fefb:6fee on port Rear(Rear("rear0"))/0
812026-05-20T17:41:11.298ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::f00c:acff:fefb:6fee on port Rear(Rear("rear0"))/0
822026-05-20T17:41:11.298ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
832026-05-20T17:41:11.298ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
84 May 20 17:41:14.775 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 858, tv_nsec: 591414980 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
85 May 20 17:41:14.776 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
86 May 20 17:41:14.779 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
87 May 20 17:41:30.388 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
88 May 20 17:41:30.389 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 20.006899855s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
89 May 20 17:41:30.575 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 873, tv_nsec: 783420793 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
90 May 20 17:41:30.576 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
91 May 20 17:41:30.577 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
922026-05-20T17:41:37.911ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::f00c:acff:fefb:6fee missed solicit interval
93 May 20 17:41:45.537 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
94 May 20 17:41:45.537 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 35.196856356s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
95 May 20 17:41:46.014 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 889, tv_nsec: 264688852 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
96 May 20 17:41:46.015 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
97 May 20 17:41:46.016 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
982026-05-20T17:41:48.205ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::f00c:acff:fefb:6fee missed solicit interval
99 May 20 17:42:01.009 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
100 May 20 17:42:01.010 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 50.680056793s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
101 May 20 17:42:02.077 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 905, tv_nsec: 336902732 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
102 May 20 17:42:02.078 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
103 May 20 17:42:02.080 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
104 May 20 17:42:17.081 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
105 May 20 17:42:17.081 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 66.750808726s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
106 May 20 17:42:18.996 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 922, tv_nsec: 255700231 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
107 May 20 17:42:18.997 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
108 May 20 17:42:18.998 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
109 May 20 17:42:33.998 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
110 May 20 17:42:33.998 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 83.669915073s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
111 May 20 17:42:39.752 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 943, tv_nsec: 13308821 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
112 May 20 17:42:39.753 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
113 May 20 17:42:39.754 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1142026-05-20T17:42:57.747ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::f00c:acff:fefb:6fee missed solicit interval
115 May 20 17:42:58.220 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
116 May 20 17:42:58.429 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 108.041842201s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
117 May 20 17:43:05.492 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 968, tv_nsec: 737099175 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
118 May 20 17:43:05.495 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
119 May 20 17:43:05.517 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
120 May 20 17:43:20.520 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
121 May 20 17:43:20.523 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 130.192774147s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
122 May 20 17:43:30.669 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 993, tv_nsec: 927850765 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
123 May 20 17:43:30.670 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
124 May 20 17:43:30.672 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
125 May 20 17:43:34.066 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::b]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
126 May 20 17:43:34.066 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): client error (Connect): tcp connect error: Connection refused (os error 146), delay: 143.734880312s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
127 May 20 17:44:20.050 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5c1793b7-78b8-4142-8ad5-b70122a07a5c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("87f9f4b2-22e6-4671-9e5f-546c037b4034.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d25b946c-4926-46d5-9030-aa14ac532172.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1043, tv_nsec: 307577848 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
128 May 20 17:44:20.051 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
129 May 20 17:44:20.052 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
130 May 20 17:44:20.113 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "a049a1a7-2691-45e5-bb0d-171158f381d0", "content-length": "41", "date": "Wed, 20 May 2026 17:44:20 GMT"} }), component: producer-registration-task
131 May 20 17:44:20.113 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
132 May 20 17:44:32.370 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:1025
133 May 20 17:44:32.378 INFO request completed, latency_us: 7685, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: d6cc9ac2-1645-4ac2-80de-b16c452d3177, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
134 May 20 17:44:33.370 INFO request completed, latency_us: 953, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: a707455f-9fc4-4263-a484-ddaa090f1d61, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
135 May 20 17:44:34.369 INFO request completed, latency_us: 607, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 54b8b1eb-68c8-4a83-9917-f477bc1c9608, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
136 May 20 17:44:35.370 INFO request completed, latency_us: 846, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 58914c48-501b-4c30-8cbb-159312c46a49, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
137 May 20 17:44:36.370 INFO request completed, latency_us: 579, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: b57016da-adc0-4425-8125-f9258fe21eb7, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
138 May 20 17:44:37.369 INFO request completed, latency_us: 583, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: d58d8652-f57e-4332-ad14-3e95091e87c2, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
139 May 20 17:44:38.370 INFO request completed, latency_us: 610, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 1983608e-d82e-4653-871f-be713471eed9, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
140 May 20 17:44:39.369 INFO request completed, latency_us: 602, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: d4972d1f-6dd6-4048-867c-c05609577ddf, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
141 May 20 17:44:40.370 INFO request completed, latency_us: 819, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 3778f239-ec9c-4801-8d1c-b20f1f5594e1, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
142 May 20 17:44:41.369 INFO request completed, latency_us: 642, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 1c95781d-6a67-404f-ac72-7d352bf0913f, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
143 May 20 17:44:42.369 INFO request completed, latency_us: 574, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: d1d72f90-631b-42a7-b7d7-e7828251e344, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
144 May 20 17:44:43.370 INFO request completed, latency_us: 915, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: ba1baef6-71fd-4dd8-a6c0-903317c071ed, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
145 May 20 17:44:44.369 INFO request completed, latency_us: 630, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 73a6e55f-2760-469c-801e-f9f7d84f465c, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
146 May 20 17:44:45.370 INFO request completed, latency_us: 632, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 63fa59c7-4ad4-4fd3-a7cd-43423bc582a1, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
147 May 20 17:44:46.370 INFO request completed, latency_us: 706, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 18009c21-9477-46c8-9b8e-de2937e8d678, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
148 May 20 17:44:47.370 INFO request completed, latency_us: 597, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: d69a019e-2cf8-404b-9996-332d457f4fc0, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
149 May 20 17:44:48.370 INFO request completed, latency_us: 1002, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 1c6fda72-c0c4-4d44-8317-ad0ca9a00d8e, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
150 May 20 17:44:49.370 INFO request completed, latency_us: 740, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: fdf81a2b-6e94-4493-b081-127c932218ea, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
151 May 20 17:44:50.370 INFO request completed, latency_us: 842, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 704df662-4a63-45d5-b9ea-e09d41c8e15c, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
152 May 20 17:44:51.370 INFO request completed, latency_us: 706, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 50630fc2-ebcd-4053-b0cb-2705f9c45406, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
153 May 20 17:44:52.369 INFO request completed, latency_us: 586, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: b66ad8a8-ce96-4386-863c-4343da0a3022, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
154 May 20 17:44:53.370 INFO request completed, latency_us: 677, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: a337f0d6-7749-49f8-bc08-a7d53e5bfd65, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
155 May 20 17:44:54.370 INFO request completed, latency_us: 818, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: f670f06e-c94c-4556-a2ec-41dfb042bbff, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
156 May 20 17:44:55.370 INFO request completed, latency_us: 697, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 211f898f-8522-4324-b3c8-9e7f79ac889d, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
157 May 20 17:44:56.370 INFO request completed, latency_us: 849, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 985ae972-cc1b-4189-8296-668c182f855b, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
158 May 20 17:44:57.369 INFO request completed, latency_us: 585, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 416f5a48-cd9a-4421-9ba8-ea4edae0a637, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
159 May 20 17:44:58.370 INFO request completed, latency_us: 716, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: fb3fb797-36de-4a49-9073-828073cf603c, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
160 May 20 17:44:59.370 INFO request completed, latency_us: 587, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 56ac5304-47a1-43da-b280-43fe742014ac, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
161 May 20 17:45:00.370 INFO request completed, latency_us: 721, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 55c04357-cf4b-49fb-851a-fe39e4e632f6, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
162 May 20 17:45:01.369 INFO request completed, latency_us: 592, response_code: 200, uri: /ef5384d3-e66d-454a-b152-dbe827806e72, method: GET, req_id: 9a41d9d9-1146-4ca2-9c4b-fbe3270e31be, remote_addr: [fd00:1122:3344:101::d]:35741, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867