Max level shown:
1 [ Jul 18 19:40:39 Disabled. ]
2 [ Jul 18 19:40:39 Rereading configuration. ]
3 [ Jul 18 19:40:40 Enabled. ]
4 [ Jul 18 19:40:40 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
5 [ Jul 18 19:40:40 Method "start" exited with status 0. ]
6 [ Jul 18 19:40:40 Rereading configuration. ]
7 [ Jul 18 19:40:40 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 215"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=215
11 + getopts c: o
12 + [[ -n 215 ]]
13 + pgrep -c 215
14 + PARENT_PID=2280
15 + [[ -n 2280 ]]
16 + pgrep -P 2280
17 + CHILD_PID=2282
18 + [[ -n 2282 ]]
19 + kill -USR1 2282
20 [ Jul 18 19:40:40 Method "refresh" exited with status 0. ]
21 [ Jul 18 19:40:40 Stopping because all processes in service exited. ]
22 [ Jul 18 19:40:40 Executing stop method (:kill). ]
23 [ Jul 18 19:40:40 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
24 [ Jul 18 19:40:40 Method "start" exited with status 0. ]
252025-07-18T19:40:40.576ZINFOslog-rs: signal handler waiting for context
262025-07-18T19:40:40.594ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
272025-07-18T19:40:40.594ZERROslog-rs: failed to start stats server: underlay address not found
282025-07-18T19:40:40.594ZINFOslog-rs: signal handler waiting got context
292025-07-18T19:40:40.595ZINFOslog-rs: admin: listening on [::1]:8000
302025-07-18T19:40:41.594ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
312025-07-18T19:40:42.594ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
322025-07-18T19:40:43.595ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
332025-07-18T19:40:44.595ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
342025-07-18T19:40:45.595ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
352025-07-18T19:40:46.595ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
362025-07-18T19:40:47.595ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
372025-07-18T19:40:48.595ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
382025-07-18T19:40:49.595ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
392025-07-18T19:40:50.595ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::cb8:54ff:fe20:2cf9 on if tfportrear0_0 index 3
402025-07-18T19:40:50.595ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
41 [ Jul 18 19:40:50 Rereading configuration. ]
42 [ Jul 18 19:40:50 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 219"). ]
43 + CONTRACT=''
44 + getopts c: o
45 + CONTRACT=219
46 + getopts c: o
47 + [[ -n 219 ]]
48 + pgrep -c 219
49 + PARENT_PID=2301
50 + [[ -n 2301 ]]
51 + pgrep -P 2301
52 + CHILD_PID=2302
53 + [[ -n 2302 ]]
54 + kill -USR1 2302
552025-07-18T19:40:51.009ZINFOslog-rs: handling smf refresh
56 [ Jul 18 19:40:51 Method "refresh" exited with status 0. ]
572025-07-18T19:40:51.011ZINFOslog-rs: starting stats server on smf refresh
58 Jul 18 19:40:51.012 DEBG registered DTrace probes
59 Jul 18 19:40:51.012 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
60 Jul 18 19:40:51.012 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.2/src/server.rs:328
61 Jul 18 19:40:51.012 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
62 Jul 18 19:40:51.013 DEBG Nexus IP not provided, will use DNS to resolve it
63 Jul 18 19:40:51.013 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/internal-dns/resolver/src/resolver.rs:111
64 Jul 18 19:40:51.013 DEBG starting producer registration task
65 Jul 18 19:40:51.013 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 8c9d2fe9-a895-4c3f-a88b-f32ffc48b0d5, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:279
66 Jul 18 19:40:51.013 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
672025-07-18T19:40:52.207ZWARNslog-rs: [tfportrear0_0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd85:76fc:6cfc:8bc8::1, width: 64 }}))
682025-07-18T19:40:52.207ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd85:76fc:6cfc:8bc8::1, vni: 99, metric: 18446744073709551615, }, }
692025-07-18T19:40:52.208ZWARNslog-rs: [tfportrear0_0] admin event in solicit state: Announce(Tunnel({TunnelOrigin { overlay_prefix: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), boundary_addr: fd85:76fc:6cfc:8bc8::1, vni: 99, metric: 18446744073709551615 }}))
702025-07-18T19:40:52.596ZINFOslog-rs: [tfportrear0_0] nbr is fe80::fcf2:e3ff:fec6:a7ca@buskin server
712025-07-18T19:40:52.596ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
722025-07-18T19:40:52.596ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::cb8:54ff:fe20:2cf9]:56797
732025-07-18T19:40:52.596ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
742025-07-18T19:40:52.598ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
752025-07-18T19:40:52.598ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
762025-07-18T19:40:52.599ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
772025-07-18T19:40:52.600ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
782025-07-18T19:40:52.600ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
792025-07-18T19:40:52.600ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
802025-07-18T19:40:52.600ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
812025-07-18T19:40:52.601ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
822025-07-18T19:40:52.601ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
83 Jul 18 19:40:56.015 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 632, tv_nsec: 581547709 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
84 Jul 18 19:40:56.016 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
85 Jul 18 19:40:56.016 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
86 Jul 18 19:41:10.375 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
87 Jul 18 19:41:10.375 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 20.004421048s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
88 Jul 18 19:41:10.559 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 647, tv_nsec: 767506095 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
89 Jul 18 19:41:10.560 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
90 Jul 18 19:41:10.560 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
91 Jul 18 19:41:25.601 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
92 Jul 18 19:41:25.601 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 35.190268862s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
93 Jul 18 19:41:25.937 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 663, tv_nsec: 104774804 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
94 Jul 18 19:41:25.937 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
95 Jul 18 19:41:25.937 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
96 Jul 18 19:41:40.979 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
97 Jul 18 19:41:40.979 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 50.526923205s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
98 Jul 18 19:41:41.591 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 678, tv_nsec: 715991908 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
99 Jul 18 19:41:41.592 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
100 Jul 18 19:41:41.592 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
101 Jul 18 19:41:56.634 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
102 Jul 18 19:41:56.634 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 66.139316417s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
103 Jul 18 19:41:58.325 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 695, tv_nsec: 404831134 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
104 Jul 18 19:41:58.326 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
105 Jul 18 19:41:58.326 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
106 Jul 18 19:42:13.369 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
107 Jul 18 19:42:13.369 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 82.828520336s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
108 Jul 18 19:42:15.445 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 712, tv_nsec: 478056255 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
109 Jul 18 19:42:15.446 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
110 Jul 18 19:42:15.446 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
111 Jul 18 19:42:30.488 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
112 Jul 18 19:42:30.488 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 99.901532593s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
113 Jul 18 19:42:38.362 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 735, tv_nsec: 333083107 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
114 Jul 18 19:42:38.363 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
115 Jul 18 19:42:38.363 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
116 Jul 18 19:42:53.405 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
117 Jul 18 19:42:53.406 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 122.756344266s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
1182025-07-18T19:43:14.863ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::fcf2:e3ff:fec6:a7ca missed solicit interval
119 Jul 18 19:43:15.997 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 772, tv_nsec: 868481327 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
120 Jul 18 19:43:15.999 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
121 Jul 18 19:43:16.006 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
122 Jul 18 19:43:31.004 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
123 Jul 18 19:43:31.005 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 160.302242793s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371
1242025-07-18T19:44:07.157ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::fcf2:e3ff:fec6:a7ca expire
1252025-07-18T19:44:07.172ZWARNslog-rs: [tfportrear0_0] expiring peer fe80::fcf2:e3ff:fec6:a7ca due to discovery event
1262025-07-18T19:44:07.172ZINFOslog-rs: [tfportrear0_0] nbr is fe80::fcf2:e3ff:fec6:a7ca@buskin server
1272025-07-18T19:44:07.173ZINFOslog-rs: [tfportrear0_0] removing routes 5 from dendrite
1282025-07-18T19:44:07.701ZERROslog-rs: [tfportrear0_0] failed to remove tunnel routes: {} OPTE driver is not attached
1292025-07-18T19:44:07.701ZDEBGslog-rs: [tfportrear0_0] redistributing expire to 0 peers
1302025-07-18T19:44:07.733ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
1312025-07-18T19:44:07.830ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::cb8:54ff:fe20:2cf9]:56797
1322025-07-18T19:44:07.879ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
1332025-07-18T19:44:07.929ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1342025-07-18T19:44:07.929ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1352025-07-18T19:44:07.947ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
1362025-07-18T19:44:07.950ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
1372025-07-18T19:44:07.950ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
1382025-07-18T19:44:07.950ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
1392025-07-18T19:44:07.951ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::fcf2:e3ff:fec6:a7ca on port Rear(Rear("rear0"))/0
1402025-07-18T19:44:07.951ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1412025-07-18T19:44:07.951ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
142 Jul 18 19:44:14.737 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("b2568d68-c2f1-4ec5-9f40-43ecf91a898a.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c0ab1674-fd55-45ce-baf5-f683df76cfe7.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("c2e32d3e-2f29-4056-a216-294e9d23f41b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(fd00:1122:3344:101::b))) }], valid_until: Instant { tv_sec: 831, tv_nsec: 604083989 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
143 Jul 18 19:44:14.744 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
144 Jul 18 19:44:14.759 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
145 Jul 18 19:44:24.206 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::c]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
146 Jul 18 19:44:24.207 WARN failed to register with Nexus, will retry, error: "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)", delay: 213.503495397s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/1d3958c/oximeter/producer/src/lib.rs:371