Max level shown:
1 [ May 4 19:27:04 Disabled. ]
2 [ May 4 19:27:04 Rereading configuration. ]
3 [ May 4 19:27:04 Enabled. ]
4 [ May 4 19:27:04 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
5 [ May 4 19:27:04 Method "start" exited with status 0. ]
6 [ May 4 19:27:04 Rereading configuration. ]
7 [ May 4 19:27:04 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 212"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=212
11 + getopts c: o
12 + [[ -n 212 ]]
13 + pgrep -c 212
14 + PARENT_PID=2283
15 + [[ -n 2283 ]]
16 + pgrep -P 2283
17 + CHILD_PID=2287
18 + [[ -n 2287 ]]
19 + kill -USR1 2287
20 [ May 4 19:27:04 Method "refresh" exited with status 0. ]
21 [ May 4 19:27:04 Stopping because all processes in service exited. ]
22 [ May 4 19:27:04 Executing stop method (:kill). ]
23 [ May 4 19:27:04 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
24 [ May 4 19:27:04 Method "start" exited with status 0. ]
252026-05-04T19:27:04.899ZINFOslog-rs: signal handler waiting for context
262026-05-04T19:27:04.920ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
272026-05-04T19:27:04.920ZERROslog-rs: failed to start stats server: underlay address not found
282026-05-04T19:27:04.920ZINFOslog-rs: signal handler waiting got context
292026-05-04T19:27:04.921ZINFOslog-rs: admin: listening on [::1]:8000
302026-05-04T19:27:04.922ZINFOslog-rs: admin: server started
312026-05-04T19:27:05.920ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
322026-05-04T19:27:06.920ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
332026-05-04T19:27:07.920ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
342026-05-04T19:27:08.920ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
352026-05-04T19:27:09.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
362026-05-04T19:27:10.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
372026-05-04T19:27:11.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
382026-05-04T19:27:12.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
392026-05-04T19:27:13.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
402026-05-04T19:27:14.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
412026-05-04T19:27:15.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
422026-05-04T19:27:16.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
432026-05-04T19:27:17.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
442026-05-04T19:27:18.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
452026-05-04T19:27:19.921ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
462026-05-04T19:27:20.922ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
472026-05-04T19:27:21.922ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
482026-05-04T19:27:22.922ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
492026-05-04T19:27:23.922ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
502026-05-04T19:27:24.922ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
512026-05-04T19:27:25.922ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::2022:adff:fea1:78fd on if tfportrear0_0 index 3
522026-05-04T19:27:25.922ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
532026-05-04T19:27:25.923ZINFOslog-rs: [tfportrear0_0] nbr is fe80::b8f1:2dff:fea4:25a5@buskin server
542026-05-04T19:27:25.923ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
552026-05-04T19:27:25.923ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::2022:adff:fea1:78fd]:56797
562026-05-04T19:27:25.923ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
57 [ May 4 19:27:26 Rereading configuration. ]
58 [ May 4 19:27:26 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 217"). ]
59 + CONTRACT=''
60 + getopts c: o
61 + CONTRACT=217
62 + getopts c: o
63 + [[ -n 217 ]]
64 + pgrep -c 217
65 + PARENT_PID=2306
66 + [[ -n 2306 ]]
67 + pgrep -P 2306
68 + CHILD_PID=2307
69 + [[ -n 2307 ]]
70 + kill -USR1 2307
712026-05-04T19:27:26.232ZINFOslog-rs: handling smf refresh
72 [ May 4 19:27:26 Method "refresh" exited with status 0. ]
732026-05-04T19:27:26.234ZINFOslog-rs: starting stats server on smf refresh
74 May 04 19:27:26.234 DEBG registered DTrace probes
75 May 04 19:27:26.235 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
76 May 04 19:27:26.235 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328
77 May 04 19:27:26.235 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
78 May 04 19:27:26.236 DEBG Nexus IP not provided, will use DNS to resolve it
79 May 04 19:27:26.236 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/internal-dns/resolver/src/resolver.rs:144
80 May 04 19:27:26.236 DEBG starting producer registration task
81 May 04 19:27:26.236 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 9e103647-c40a-4413-9353-c8d59ca4f380, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:283
82 May 04 19:27:26.236 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
83 May 04 19:27:26.237 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 659, tv_nsec: 898418404 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
84 May 04 19:27:26.238 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
85 May 04 19:27:26.240 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
862026-05-04T19:27:27.470ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd8d:d938:e961:e65e::1, vni: 99, metric: 18446744073709551615, }, }
872026-05-04T19:27:27.925ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
882026-05-04T19:27:27.926ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
892026-05-04T19:27:27.926ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
902026-05-04T19:27:27.926ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
912026-05-04T19:27:27.927ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
922026-05-04T19:27:27.928ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
932026-05-04T19:27:27.928ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
942026-05-04T19:27:27.928ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
952026-05-04T19:27:27.928ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
962026-05-04T19:27:27.928ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
97 May 04 19:27:41.365 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
98 May 04 19:27:41.365 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 15.004731523s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
99 May 04 19:27:41.582 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 675, tv_nsec: 120005059 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
100 May 04 19:27:41.583 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
101 May 04 19:27:41.584 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1022026-05-04T19:27:42.048ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::b8f1:2dff:fea4:25a5 missed solicit interval
1032026-05-04T19:27:44.049ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::b8f1:2dff:fea4:25a5 missed solicit interval
1042026-05-04T19:27:46.050ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::b8f1:2dff:fea4:25a5 missed solicit interval
1052026-05-04T19:27:48.050ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::b8f1:2dff:fea4:25a5 missed solicit interval
106 May 04 19:27:56.544 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
107 May 04 19:27:56.544 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 30.225282548s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
108 May 04 19:27:56.889 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 690, tv_nsec: 469157170 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
109 May 04 19:27:56.890 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
110 May 04 19:27:56.891 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1112026-05-04T19:28:02.109ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::b8f1:2dff:fea4:25a5 missed solicit interval
1122026-05-04T19:28:07.969ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::b8f1:2dff:fea4:25a5 expire
1132026-05-04T19:28:07.994ZINFOslog-rs: [tfportrear0_0] nbr is fe80::b8f1:2dff:fea4:25a5@buskin server
1142026-05-04T19:28:07.994ZWARNslog-rs: [tfportrear0_0] expiring peer fe80::b8f1:2dff:fea4:25a5 due to discovery event
1152026-05-04T19:28:07.994ZINFOslog-rs: [tfportrear0_0] removing routes 5 from dendrite
1162026-05-04T19:28:08.107ZERROslog-rs: [tfportrear0_0] failed to remove tunnel routes: {} OPTE driver is not attached
1172026-05-04T19:28:08.107ZDEBGslog-rs: [tfportrear0_0] redistributing expire to 0 peers
1182026-05-04T19:28:08.108ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
1192026-05-04T19:28:08.121ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::2022:adff:fea1:78fd]:56797
1202026-05-04T19:28:08.126ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
1212026-05-04T19:28:08.149ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1222026-05-04T19:28:08.149ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1232026-05-04T19:28:08.153ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
1242026-05-04T19:28:08.154ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
1252026-05-04T19:28:08.154ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
1262026-05-04T19:28:08.154ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
1272026-05-04T19:28:08.155ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::b8f1:2dff:fea4:25a5 on port Rear(Rear("rear0"))/0
1282026-05-04T19:28:08.155ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1292026-05-04T19:28:08.155ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
130 May 04 19:28:11.859 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
131 May 04 19:28:11.861 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 45.582950211s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
132 May 04 19:28:13.030 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 706, tv_nsec: 650779974 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
133 May 04 19:28:13.031 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
134 May 04 19:28:13.037 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
135 May 04 19:28:27.997 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
136 May 04 19:28:27.997 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 61.76373896s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
137 May 04 19:28:30.606 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 724, tv_nsec: 277479356 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
138 May 04 19:28:30.607 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
139 May 04 19:28:30.608 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
140 May 04 19:28:45.569 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
141 May 04 19:28:45.569 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 79.382851717s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
142 May 04 19:28:48.696 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 742, tv_nsec: 415908307 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
143 May 04 19:28:48.696 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
144 May 04 19:28:48.697 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
145 May 04 19:29:03.663 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
146 May 04 19:29:03.665 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 97.528020883s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
147 May 04 19:29:12.006 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 765, tv_nsec: 788148487 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
148 May 04 19:29:12.006 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
149 May 04 19:29:12.010 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
150 May 04 19:29:27.008 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
151 May 04 19:29:27.009 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 120.898617821s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379
152 May 04 19:29:37.359 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("6a0611e6-38d2-4444-bc5c-513ef79356de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("88b09bf5-c38f-4ac8-892e-55b03757b0ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("f89597bb-ad49-4d62-8cd6-c4c977aaf9e2.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 791, tv_nsec: 145757211 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
153 May 04 19:29:37.360 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
154 May 04 19:29:37.361 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
155 May 04 19:29:37.364 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::c]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
156 May 04 19:29:37.365 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): client error (Connect): tcp connect error: Connection refused (os error 146), delay: 131.254071203s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/dfcd9ac/oximeter/producer/src/lib.rs:379