Max level shown:
1 [ Mar 20 19:26:26 Disabled. ]
2 [ Mar 20 19:26:26 Rereading configuration. ]
3 [ Mar 20 19:26:26 Enabled. ]
4 [ Mar 20 19:26:26 Rereading configuration. ]
5 [ Mar 20 19:26:26 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Mar 20 19:26:26 Method "start" exited with status 0. ]
72026-03-20T19:26:26.846ZINFOslog-rs: signal handler waiting for context
82026-03-20T19:26:26.867ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-03-20T19:26:26.867ZERROslog-rs: failed to start stats server: underlay address not found
102026-03-20T19:26:26.867ZINFOslog-rs: signal handler waiting got context
112026-03-20T19:26:26.868ZINFOslog-rs: admin: listening on [::1]:8000
122026-03-20T19:26:26.868ZINFOslog-rs: admin: server started
132026-03-20T19:26:27.867ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-03-20T19:26:28.867ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-03-20T19:26:29.867ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-03-20T19:26:30.867ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-03-20T19:26:31.867ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-03-20T19:26:32.867ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-03-20T19:26:33.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-03-20T19:26:34.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-03-20T19:26:35.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-03-20T19:26:36.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
232026-03-20T19:26:37.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
242026-03-20T19:26:38.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
252026-03-20T19:26:39.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
262026-03-20T19:26:40.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
272026-03-20T19:26:41.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
282026-03-20T19:26:42.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
292026-03-20T19:26:43.868ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
302026-03-20T19:26:44.869ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
312026-03-20T19:26:45.869ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
322026-03-20T19:26:46.869ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
332026-03-20T19:26:47.869ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::bc73:fdff:fe28:644e on if tfportrear0_0 index 3
342026-03-20T19:26:47.869ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
352026-03-20T19:26:47.869ZINFOslog-rs: [tfportrear0_0] nbr is fe80::a8ff:dfff:fe4c:f5e6@sock server
362026-03-20T19:26:47.869ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
372026-03-20T19:26:47.870ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::bc73:fdff:fe28:644e]:56797
382026-03-20T19:26:47.870ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
39 [ Mar 20 19:26:48 Rereading configuration. ]
40 [ Mar 20 19:26:48 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 212"). ]
41 + CONTRACT=''
42 + getopts c: o
43 + CONTRACT=212
44 + getopts c: o
45 + [[ -n 212 ]]
46 + pgrep -c 212
47 + PARENT_PID=2284
48 + [[ -n 2284 ]]
49 + pgrep -P 2284
50 + CHILD_PID=2286
51 + [[ -n 2286 ]]
52 + kill -USR1 2286
532026-03-20T19:26:48.246ZINFOslog-rs: handling smf refresh
54 [ Mar 20 19:26:48 Method "refresh" exited with status 0. ]
552026-03-20T19:26:48.249ZINFOslog-rs: starting stats server on smf refresh
56 Mar 20 19:26:48.250 DEBG registered DTrace probes
57 Mar 20 19:26:48.250 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
58 Mar 20 19:26:48.250 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:328
59 Mar 20 19:26:48.251 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
60 Mar 20 19:26:48.251 DEBG Nexus IP not provided, will use DNS to resolve it
61 Mar 20 19:26:48.251 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/internal-dns/resolver/src/resolver.rs:144
62 Mar 20 19:26:48.251 DEBG starting producer registration task
63 Mar 20 19:26:48.251 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 33817fde-0630-4cce-aeab-497dd0fcae1c, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:281
64 Mar 20 19:26:48.251 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
65 Mar 20 19:26:48.252 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 1.046246ms, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
66 Mar 20 19:26:48.417 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 166.61074ms, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
67 Mar 20 19:26:49.018 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("064f1755-0a55-4fca-9bd7-ec0ae205759d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("231b1ff8-d1d2-483b-86eb-0125bc15f1e9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 659, tv_nsec: 653506528 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
68 Mar 20 19:26:49.019 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
69 Mar 20 19:26:49.019 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
702026-03-20T19:26:49.451ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdee:887e:741f:7873::1, vni: 99, metric: 18446744073709551615, }, }
712026-03-20T19:26:49.871ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
722026-03-20T19:26:49.872ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
732026-03-20T19:26:49.872ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
742026-03-20T19:26:49.873ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
752026-03-20T19:26:49.874ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
762026-03-20T19:26:49.874ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
772026-03-20T19:26:49.874ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
782026-03-20T19:26:49.874ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
792026-03-20T19:26:49.874ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
802026-03-20T19:26:49.874ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
812026-03-20T19:27:03.870ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
82 Mar 20 19:27:04.017 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
83 Mar 20 19:27:04.017 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 15.76910193s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
84 Mar 20 19:27:04.974 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("064f1755-0a55-4fca-9bd7-ec0ae205759d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("231b1ff8-d1d2-483b-86eb-0125bc15f1e9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 675, tv_nsec: 615769031 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
85 Mar 20 19:27:04.975 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
86 Mar 20 19:27:04.975 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
872026-03-20T19:27:05.870ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
882026-03-20T19:27:07.870ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
892026-03-20T19:27:09.870ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
902026-03-20T19:27:11.870ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
912026-03-20T19:27:13.870ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
922026-03-20T19:27:15.870ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
932026-03-20T19:27:17.871ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
942026-03-20T19:27:19.871ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
95 Mar 20 19:27:19.934 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
96 Mar 20 19:27:19.934 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 31.731450844s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
972026-03-20T19:27:21.871ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
98 Mar 20 19:27:22.167 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("064f1755-0a55-4fca-9bd7-ec0ae205759d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("231b1ff8-d1d2-483b-86eb-0125bc15f1e9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 692, tv_nsec: 857087182 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
99 Mar 20 19:27:22.168 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
100 Mar 20 19:27:22.168 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1012026-03-20T19:27:23.872ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
1022026-03-20T19:27:25.873ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
1032026-03-20T19:27:27.874ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
1042026-03-20T19:27:29.875ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
1052026-03-20T19:27:31.876ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
1062026-03-20T19:27:33.877ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
1072026-03-20T19:27:38.936ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 expire
1082026-03-20T19:27:38.936ZWARNslog-rs: [tfportrear0_0] expiring peer fe80::a8ff:dfff:fe4c:f5e6 due to discovery event
1092026-03-20T19:27:38.936ZINFOslog-rs: [tfportrear0_0] removing routes 5 from dendrite
1102026-03-20T19:27:38.936ZINFOslog-rs: [tfportrear0_0] nbr is fe80::a8ff:dfff:fe4c:f5e6@sock server
111 Mar 20 19:27:38.989 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
112 Mar 20 19:27:38.989 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 50.828072424s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
1132026-03-20T19:27:39.051ZERROslog-rs: [tfportrear0_0] failed to remove tunnel routes: {} OPTE driver is not attached
1142026-03-20T19:27:39.051ZDEBGslog-rs: [tfportrear0_0] redistributing expire to 0 peers
1152026-03-20T19:27:39.051ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
1162026-03-20T19:27:39.051ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::bc73:fdff:fe28:644e]:56797
1172026-03-20T19:27:39.059ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
1182026-03-20T19:27:39.067ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1192026-03-20T19:27:39.067ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1202026-03-20T19:27:39.075ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
1212026-03-20T19:27:39.077ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
1222026-03-20T19:27:39.077ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
1232026-03-20T19:27:39.078ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
1242026-03-20T19:27:39.078ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::a8ff:dfff:fe4c:f5e6 on port Rear(Rear("rear0"))/0
1252026-03-20T19:27:39.078ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1262026-03-20T19:27:39.078ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
127 Mar 20 19:27:44.039 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("064f1755-0a55-4fca-9bd7-ec0ae205759d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("231b1ff8-d1d2-483b-86eb-0125bc15f1e9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 714, tv_nsec: 764726827 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
128 Mar 20 19:27:44.041 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
129 Mar 20 19:27:44.041 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
130 Mar 20 19:27:59.045 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
131 Mar 20 19:27:59.045 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 70.883480245s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
132 Mar 20 19:28:10.398 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("064f1755-0a55-4fca-9bd7-ec0ae205759d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("231b1ff8-d1d2-483b-86eb-0125bc15f1e9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 741, tv_nsec: 122175927 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
133 Mar 20 19:28:10.399 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
134 Mar 20 19:28:10.400 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1352026-03-20T19:28:19.379ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::a8ff:dfff:fe4c:f5e6 missed solicit interval
136 Mar 20 19:28:25.401 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
137 Mar 20 19:28:25.401 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 97.239237271s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
138 Mar 20 19:28:36.882 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("064f1755-0a55-4fca-9bd7-ec0ae205759d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("231b1ff8-d1d2-483b-86eb-0125bc15f1e9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 767, tv_nsec: 606419506 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
139 Mar 20 19:28:36.882 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
140 Mar 20 19:28:36.883 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
141 Mar 20 19:28:51.884 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
142 Mar 20 19:28:51.884 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 123.722573476s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/79fac7d/oximeter/producer/src/lib.rs:377
143 Mar 20 19:29:35.620 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("064f1755-0a55-4fca-9bd7-ec0ae205759d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("231b1ff8-d1d2-483b-86eb-0125bc15f1e9.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("743fbd61-3fb3-4e94-b43b-dee0b2b34790.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 826, tv_nsec: 344210398 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
144 Mar 20 19:29:35.621 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
145 Mar 20 19:29:35.621 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
146 Mar 20 19:29:35.632 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "290b9e44-c1de-42f6-8c7a-bfab4a4db206", "content-length": "41", "date": "Fri, 20 Mar 2026 19:29:35 GMT"} }), component: producer-registration-task
147 Mar 20 19:29:35.632 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
148 Mar 20 19:29:39.455 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:1025
149 Mar 20 19:29:39.465 INFO request completed, latency_us: 9911, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 5b0e69d1-9f8d-4d70-aab0-4f0a3cf89ffb, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
150 Mar 20 19:29:40.455 INFO request completed, latency_us: 902, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: b66f743f-b39b-4734-93e6-d0911f99327b, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
151 Mar 20 19:29:41.455 INFO request completed, latency_us: 1029, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 3bbe1768-77cc-471e-9b98-27c702fddb12, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
152 Mar 20 19:29:42.455 INFO request completed, latency_us: 687, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 6b287af3-c4d0-4fa9-b204-c0a7ce3ab8ff, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
153 Mar 20 19:29:43.455 INFO request completed, latency_us: 619, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 1b2bbdf9-e672-4692-a686-51698aa71663, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
154 Mar 20 19:29:44.455 INFO request completed, latency_us: 957, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: bbfc50f0-383a-4329-b7b7-fb1a77251cf8, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
155 Mar 20 19:29:45.455 INFO request completed, latency_us: 749, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 6bc916a0-8cd5-4e9c-8ec4-a49d06ad1d44, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
156 Mar 20 19:29:46.455 INFO request completed, latency_us: 707, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: cb4207e3-6240-4226-b6ea-74ba185190a4, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
157 Mar 20 19:29:47.455 INFO request completed, latency_us: 731, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 73a547c4-5d91-4d61-8002-70d179dfa148, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
158 Mar 20 19:29:48.455 INFO request completed, latency_us: 608, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 8f40f9ff-f4f3-4be4-929a-02a82d12fb96, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
159 Mar 20 19:29:49.455 INFO request completed, latency_us: 637, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 17e21268-ed4b-42e2-9633-e9af205aa39b, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
160 Mar 20 19:29:50.455 INFO request completed, latency_us: 730, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: ebb7f86a-ce3b-40f0-bd62-f4f2398ac16b, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
161 Mar 20 19:29:51.456 INFO request completed, latency_us: 995, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 2dbf43b0-5a8c-4482-a947-5aa8577f3258, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
162 Mar 20 19:29:52.455 INFO request completed, latency_us: 707, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 20e2accc-1da8-4600-a83c-38c9ec3745b7, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
163 Mar 20 19:29:53.455 INFO request completed, latency_us: 959, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 8bddac1b-b036-4b7e-8f0b-75adb07ddc35, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
164 Mar 20 19:29:54.455 INFO request completed, latency_us: 607, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: cb9759e8-849c-458a-9c9d-db53b2ec1cbb, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
165 Mar 20 19:29:55.455 INFO request completed, latency_us: 611, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: d4e91081-1696-40c9-8d25-80ea904625c7, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
166 Mar 20 19:29:56.455 INFO request completed, latency_us: 617, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 0ba7171e-b7cf-4194-a032-83c15bdc1954, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
167 Mar 20 19:29:57.455 INFO request completed, latency_us: 606, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 7567b606-f251-4fec-be28-4f67e8413a6c, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
168 Mar 20 19:29:58.455 INFO request completed, latency_us: 885, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 10c4338c-2072-417d-86ff-547c4cda807f, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
169 Mar 20 19:29:59.454 INFO request completed, latency_us: 616, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: 6a07e887-4bbc-4de3-aec3-4a1b09d35085, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867
170 Mar 20 19:30:00.455 INFO request completed, latency_us: 1085, response_code: 200, uri: /33817fde-0630-4cce-aeab-497dd0fcae1c, method: GET, req_id: f9ae35a6-d6a8-4954-a6e5-40038e62dc9d, remote_addr: [fd00:1122:3344:101::d]:39473, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:867