Max level shown:
1 [ Sep 7 00:41:25 Disabled. ]
2 [ Sep 7 00:41:25 Rereading configuration. ]
3 [ Sep 7 00:41:26 Enabled. ]
4 [ Sep 7 00:41:26 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
5 [ Sep 7 00:41:26 Method "start" exited with status 0. ]
6 [ Sep 7 00:41:26 Rereading configuration. ]
7 [ Sep 7 00:41:26 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 212"). ]
8 + CONTRACT=''
9 + getopts c: o
10 + CONTRACT=212
11 + getopts c: o
12 + [[ -n 212 ]]
13 + pgrep -c 212
14 + PARENT_PID=2288
15 + [[ -n 2288 ]]
16 + pgrep -P 2288
17 + CHILD_PID=2291
18 + [[ -n 2291 ]]
19 + kill -USR1 2291
20 [ Sep 7 00:41:26 Method "refresh" exited with status 0. ]
21 [ Sep 7 00:41:26 Stopping because all processes in service exited. ]
22 [ Sep 7 00:41:26 Executing stop method (:kill). ]
23 [ Sep 7 00:41:26 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
24 [ Sep 7 00:41:26 Method "start" exited with status 0. ]
252026-09-07T00:41:26.668ZINFOslog-rs: signal handler waiting for context
262026-09-07T00:41:26.690ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
272026-09-07T00:41:26.690ZERROslog-rs: failed to start stats server: underlay address not found
282026-09-07T00:41:26.690ZINFOslog-rs: signal handler waiting got context
292026-09-07T00:41:26.691ZINFOslog-rs: admin: listening on [::1]:8000
302026-09-07T00:41:27.690ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
312026-09-07T00:41:28.690ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
322026-09-07T00:41:29.690ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
332026-09-07T00:41:30.690ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
342026-09-07T00:41:31.690ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
352026-09-07T00:41:32.700ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
362026-09-07T00:41:33.700ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
372026-09-07T00:41:34.701ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
382026-09-07T00:41:35.701ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
392026-09-07T00:41:36.701ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
402026-09-07T00:41:37.701ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::8f9:32ff:fe3b:1eee on if tfportrear0_0 index 3
412026-09-07T00:41:37.701ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
422026-09-07T00:41:37.702ZINFOslog-rs: [tfportrear0_0] nbr is fe80::6c07:fbff:fe35:711d@sock server
432026-09-07T00:41:37.702ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
442026-09-07T00:41:37.702ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::8f9:32ff:fe3b:1eee]:56797
452026-09-07T00:41:37.702ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
46 [ Sep 7 00:41:38 Rereading configuration. ]
47 [ Sep 7 00:41:38 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 217"). ]
48 + CONTRACT=''
49 + getopts c: o
50 + CONTRACT=217
51 + getopts c: o
52 + [[ -n 217 ]]
53 + pgrep -c 217
54 + PARENT_PID=2309
55 + [[ -n 2309 ]]
56 + pgrep -P 2309
57 + CHILD_PID=2310
58 + [[ -n 2310 ]]
59 + kill -USR1 2310
602026-09-07T00:41:38.236ZINFOslog-rs: handling smf refresh
61 [ Sep 7 00:41:38 Method "refresh" exited with status 0. ]
622026-09-07T00:41:38.238ZINFOslog-rs: starting stats server on smf refresh
63 Sep 07 00:41:38.239 DEBG registered DTrace probes
64 Sep 07 00:41:38.239 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
65 Sep 07 00:41:38.239 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
66 Sep 07 00:41:38.239 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
67 Sep 07 00:41:38.240 DEBG Nexus IP not provided, will use DNS to resolve it
68 Sep 07 00:41:38.240 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
69 Sep 07 00:41:38.240 DEBG starting producer registration task
70 Sep 07 00:41:38.240 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: f8f9c0c3-a8c1-44dd-bb20-00b792b5a84d, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
71 Sep 07 00:41:38.240 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
722026-09-07T00:41:38.975ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdee:5a17:41a8:3617::1, vni: 99, metric: 18446744073709551615, }, }
732026-09-07T00:41:39.704ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
742026-09-07T00:41:39.706ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
752026-09-07T00:41:39.706ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
762026-09-07T00:41:39.708ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::6c07:fbff:fe35:711d on port Rear(Rear("rear0"))/0
772026-09-07T00:41:39.709ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::6c07:fbff:fe35:711d on port Rear(Rear("rear0"))/0
782026-09-07T00:41:39.710ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::6c07:fbff:fe35:711d on port Rear(Rear("rear0"))/0
792026-09-07T00:41:39.710ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::6c07:fbff:fe35:711d on port Rear(Rear("rear0"))/0
802026-09-07T00:41:39.710ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::6c07:fbff:fe35:711d on port Rear(Rear("rear0"))/0
812026-09-07T00:41:39.711ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
822026-09-07T00:41:39.711ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
83 Sep 07 00:41:43.243 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 560, tv_nsec: 9068210 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
84 Sep 07 00:41:43.243 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
85 Sep 07 00:41:43.245 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
862026-09-07T00:41:56.422ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
872026-09-07T00:41:58.422ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
88 Sep 07 00:41:58.955 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
89 Sep 07 00:41:58.955 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 20.00601238s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
90 Sep 07 00:41:59.295 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 575, tv_nsec: 352672927 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
91 Sep 07 00:41:59.295 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
92 Sep 07 00:41:59.296 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
932026-09-07T00:42:00.422ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
942026-09-07T00:42:02.422ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
952026-09-07T00:42:04.422ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
962026-09-07T00:42:06.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
972026-09-07T00:42:08.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
982026-09-07T00:42:10.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
992026-09-07T00:42:12.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
100 Sep 07 00:42:14.257 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
101 Sep 07 00:42:14.257 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 35.349195289s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1022026-09-07T00:42:14.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
103 Sep 07 00:42:14.783 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 590, tv_nsec: 883232099 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
104 Sep 07 00:42:14.784 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
105 Sep 07 00:42:14.785 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1062026-09-07T00:42:16.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1072026-09-07T00:42:18.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1082026-09-07T00:42:20.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1092026-09-07T00:42:22.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1102026-09-07T00:42:24.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1112026-09-07T00:42:26.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1122026-09-07T00:42:28.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
113 Sep 07 00:42:29.745 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
114 Sep 07 00:42:29.746 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 50.879727959s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1152026-09-07T00:42:30.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
116 Sep 07 00:42:30.582 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 606, tv_nsec: 725220893 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
117 Sep 07 00:42:30.583 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
118 Sep 07 00:42:30.584 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1192026-09-07T00:42:32.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1202026-09-07T00:42:34.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1212026-09-07T00:42:36.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1222026-09-07T00:42:38.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1232026-09-07T00:42:40.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1242026-09-07T00:42:42.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1252026-09-07T00:42:44.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
126 Sep 07 00:42:45.544 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
127 Sep 07 00:42:45.544 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 66.721504689s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1282026-09-07T00:42:46.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
129 Sep 07 00:42:47.618 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 623, tv_nsec: 807013843 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
130 Sep 07 00:42:47.618 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
131 Sep 07 00:42:47.619 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1322026-09-07T00:42:48.423ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1332026-09-07T00:42:50.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1342026-09-07T00:42:52.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1352026-09-07T00:42:54.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1362026-09-07T00:42:56.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1372026-09-07T00:42:58.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1382026-09-07T00:43:00.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1392026-09-07T00:43:02.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
140 Sep 07 00:43:02.580 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
141 Sep 07 00:43:02.580 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 83.803499976s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1422026-09-07T00:43:04.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
143 Sep 07 00:43:04.767 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 641, tv_nsec: 2607711 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
144 Sep 07 00:43:04.767 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
145 Sep 07 00:43:04.768 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1462026-09-07T00:43:06.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1472026-09-07T00:43:08.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1482026-09-07T00:43:10.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1492026-09-07T00:43:12.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1502026-09-07T00:43:14.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1512026-09-07T00:43:16.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1522026-09-07T00:43:18.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
153 Sep 07 00:43:19.728 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
154 Sep 07 00:43:19.729 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 100.998354166s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1552026-09-07T00:43:20.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1562026-09-07T00:43:22.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1572026-09-07T00:43:24.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1582026-09-07T00:43:26.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1592026-09-07T00:43:28.424ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
160 Sep 07 00:43:28.865 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 665, tv_nsec: 166093379 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
161 Sep 07 00:43:28.865 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
162 Sep 07 00:43:28.866 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1632026-09-07T00:43:30.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1642026-09-07T00:43:32.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1652026-09-07T00:43:34.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1662026-09-07T00:43:36.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1672026-09-07T00:43:38.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1682026-09-07T00:43:40.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
1692026-09-07T00:43:42.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
170 Sep 07 00:43:43.828 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
171 Sep 07 00:43:43.828 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 125.162996067s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1722026-09-07T00:43:44.425ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval
173 Sep 07 00:44:07.236 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("7791f2c1-888d-4d6e-bb8a-35cae3ac4510.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91fd3060-cb0f-4751-b116-8256a5a7c16e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e84a7e33-fea7-47de-ac59-8510454484de.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 703, tv_nsec: 640911245 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
174 Sep 07 00:44:07.236 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
175 Sep 07 00:44:07.237 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
176 Sep 07 00:44:22.198 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
177 Sep 07 00:44:22.198 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 163.637713852s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1782026-09-07T00:44:36.442ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::6c07:fbff:fe35:711d missed solicit interval