Max level shown:
1 [ Aug 25 09:26:00 Disabled. ]
2 [ Aug 25 09:26:00 Rereading configuration. ]
3 [ Aug 25 09:26:00 Enabled. ]
4 [ Aug 25 09:26:00 Rereading configuration. ]
5 [ Aug 25 09:26:01 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Aug 25 09:26:01 Method "start" exited with status 0. ]
72026-08-25T09:26:01.083ZINFOslog-rs: signal handler waiting for context
82026-08-25T09:26:01.100ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-08-25T09:26:01.100ZERROslog-rs: failed to start stats server: underlay address not found
102026-08-25T09:26:01.100ZINFOslog-rs: signal handler waiting got context
112026-08-25T09:26:01.102ZINFOslog-rs: admin: listening on [::]:8000
122026-08-25T09:26:02.100ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-08-25T09:26:03.101ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-08-25T09:26:04.101ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-08-25T09:26:05.101ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-08-25T09:26:06.103ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-08-25T09:26:07.103ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-08-25T09:26:08.103ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-08-25T09:26:09.103ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-08-25T09:26:10.103ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-08-25T09:26:11.103ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-08-25T09:26:12.103ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::cc22:81ff:fe8a:a07 on if tfportrear0_0 index 3
232026-08-25T09:26:12.103ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
242026-08-25T09:26:12.104ZINFOslog-rs: [tfportrear0_0] nbr is fe80::e4a1:b6ff:fee4:954a@buskin server
252026-08-25T09:26:12.104ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
262026-08-25T09:26:12.104ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::cc22:81ff:fe8a:a07]:56797
272026-08-25T09:26:12.105ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
282026-08-25T09:26:12.106ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
29 [ Aug 25 09:26:12 Rereading configuration. ]
30 [ Aug 25 09:26:12 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
31 + CONTRACT=''
32 + getopts c: o
33 + CONTRACT=213
34 + getopts c: o
35 + [[ -n 213 ]]
36 + pgrep -c 213
37 + PARENT_PID=2294
38 + [[ -n 2294 ]]
39 + pgrep -P 2294
40 + CHILD_PID=2296
41 + [[ -n 2296 ]]
42 + kill -USR1 2296
432026-08-25T09:26:12.575ZINFOslog-rs: handling smf refresh
442026-08-25T09:26:12.577ZINFOslog-rs: starting stats server on smf refresh
45 [ Aug 25 09:26:12 Method "refresh" exited with status 0. ]
46 Aug 25 09:26:12.584 DEBG registered DTrace probes
47 Aug 25 09:26:12.584 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
48 Aug 25 09:26:12.585 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
49 Aug 25 09:26:12.585 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
50 Aug 25 09:26:12.585 DEBG Nexus IP not provided, will use DNS to resolve it
51 Aug 25 09:26:12.585 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
52 Aug 25 09:26:12.585 DEBG starting producer registration task
53 Aug 25 09:26:12.585 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: fe6f0c88-0ade-4eed-8d21-07b8d5b55aed, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
54 Aug 25 09:26:12.585 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
552026-08-25T09:26:14.107ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
562026-08-25T09:26:14.108ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
572026-08-25T09:26:14.109ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
582026-08-25T09:26:14.110ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
592026-08-25T09:26:14.111ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
602026-08-25T09:26:14.111ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
612026-08-25T09:26:14.111ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
622026-08-25T09:26:14.112ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
632026-08-25T09:26:14.113ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
642026-08-25T09:26:14.659ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd9a:306e:6720:3c93::1, vni: 99, metric: 18446744073709551615, }, }
65 Aug 25 09:26:17.587 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 5.001647502s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
66 Aug 25 09:26:17.746 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2324, tv_nsec: 625530611 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
67 Aug 25 09:26:17.747 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
68 Aug 25 09:26:17.748 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
692026-08-25T09:26:22.108ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
702026-08-25T09:26:22.108ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
712026-08-25T09:26:22.109ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
722026-08-25T09:26:22.109ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
732026-08-25T09:26:22.110ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
742026-08-25T09:26:22.110ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
752026-08-25T09:26:22.110ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
762026-08-25T09:26:22.110ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
772026-08-25T09:26:29.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
782026-08-25T09:26:31.451ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
792026-08-25T09:26:31.451ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
802026-08-25T09:26:31.452ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
812026-08-25T09:26:31.453ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
822026-08-25T09:26:31.453ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
832026-08-25T09:26:31.453ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
842026-08-25T09:26:31.453ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
852026-08-25T09:26:31.454ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
862026-08-25T09:26:31.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
87 Aug 25 09:26:32.091 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
88 Aug 25 09:26:32.091 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 20.163660656s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
89 Aug 25 09:26:32.510 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2340, tv_nsec: 48908186 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
90 Aug 25 09:26:32.511 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
91 Aug 25 09:26:32.512 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
922026-08-25T09:26:33.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
932026-08-25T09:26:35.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
942026-08-25T09:26:37.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
952026-08-25T09:26:39.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
962026-08-25T09:26:41.452ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
972026-08-25T09:26:41.452ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
982026-08-25T09:26:41.453ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
992026-08-25T09:26:41.454ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1002026-08-25T09:26:41.454ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1012026-08-25T09:26:41.454ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1022026-08-25T09:26:41.454ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1032026-08-25T09:26:41.455ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1042026-08-25T09:26:41.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1052026-08-25T09:26:43.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1062026-08-25T09:26:45.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1072026-08-25T09:26:47.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
108 Aug 25 09:26:47.474 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
109 Aug 25 09:26:47.474 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 35.588060376s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
110 Aug 25 09:26:48.554 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2356, tv_nsec: 135538966 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
111 Aug 25 09:26:48.554 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
112 Aug 25 09:26:48.555 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1132026-08-25T09:26:49.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1142026-08-25T09:26:51.453ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1152026-08-25T09:26:51.453ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1162026-08-25T09:26:51.454ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1172026-08-25T09:26:51.455ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1182026-08-25T09:26:51.455ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1192026-08-25T09:26:51.455ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1202026-08-25T09:26:51.455ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1212026-08-25T09:26:51.455ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1222026-08-25T09:26:51.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1232026-08-25T09:26:53.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1242026-08-25T09:26:55.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1252026-08-25T09:26:57.459ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1262026-08-25T09:26:59.460ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1272026-08-25T09:27:01.455ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1282026-08-25T09:27:01.455ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1292026-08-25T09:27:01.455ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1302026-08-25T09:27:01.456ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1312026-08-25T09:27:01.456ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1322026-08-25T09:27:01.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1332026-08-25T09:27:01.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1342026-08-25T09:27:01.457ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1352026-08-25T09:27:01.461ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1362026-08-25T09:27:03.461ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
137 Aug 25 09:27:03.516 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
138 Aug 25 09:27:03.516 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 51.673826014s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1392026-08-25T09:27:05.461ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
140 Aug 25 09:27:05.938 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2373, tv_nsec: 567297277 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
141 Aug 25 09:27:05.939 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
142 Aug 25 09:27:05.940 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1432026-08-25T09:27:07.461ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1442026-08-25T09:27:09.461ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1452026-08-25T09:27:11.455ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1462026-08-25T09:27:11.455ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1472026-08-25T09:27:11.456ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1482026-08-25T09:27:11.456ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1492026-08-25T09:27:11.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1502026-08-25T09:27:11.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1512026-08-25T09:27:11.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1522026-08-25T09:27:11.457ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1532026-08-25T09:27:11.462ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1542026-08-25T09:27:13.462ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1552026-08-25T09:27:15.462ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1562026-08-25T09:27:17.462ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
1572026-08-25T09:27:19.462ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
158 Aug 25 09:27:20.901 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
159 Aug 25 09:27:20.901 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 69.10589697s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1602026-08-25T09:27:21.456ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1612026-08-25T09:27:21.456ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1622026-08-25T09:27:21.457ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1632026-08-25T09:27:21.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1642026-08-25T09:27:21.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1652026-08-25T09:27:21.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1662026-08-25T09:27:21.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1672026-08-25T09:27:21.458ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
168 Aug 25 09:27:24.068 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2391, tv_nsec: 745415974 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
169 Aug 25 09:27:24.068 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
170 Aug 25 09:27:24.069 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1712026-08-25T09:27:31.456ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1722026-08-25T09:27:31.456ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1732026-08-25T09:27:31.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1742026-08-25T09:27:31.457ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1752026-08-25T09:27:31.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1762026-08-25T09:27:31.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1772026-08-25T09:27:31.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1782026-08-25T09:27:31.458ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
179 Aug 25 09:27:39.054 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
180 Aug 25 09:27:39.054 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 87.283742513s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1812026-08-25T09:27:41.456ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1822026-08-25T09:27:41.456ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1832026-08-25T09:27:41.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1842026-08-25T09:27:41.458ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1852026-08-25T09:27:41.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1862026-08-25T09:27:41.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1872026-08-25T09:27:41.458ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1882026-08-25T09:27:41.458ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
189 Aug 25 09:27:43.988 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2411, tv_nsec: 682286870 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
190 Aug 25 09:27:43.989 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
191 Aug 25 09:27:43.990 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1922026-08-25T09:27:51.458ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1932026-08-25T09:27:51.458ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1942026-08-25T09:27:51.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1952026-08-25T09:27:51.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1962026-08-25T09:27:51.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1972026-08-25T09:27:51.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1982026-08-25T09:27:51.460ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
1992026-08-25T09:27:51.460ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
200 Aug 25 09:27:58.992 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
201 Aug 25 09:27:58.992 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 107.220933512s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
2022026-08-25T09:28:01.458ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2032026-08-25T09:28:01.458ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2042026-08-25T09:28:01.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2052026-08-25T09:28:01.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2062026-08-25T09:28:01.460ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2072026-08-25T09:28:01.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2082026-08-25T09:28:01.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2092026-08-25T09:28:01.460ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
2102026-08-25T09:28:11.458ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2112026-08-25T09:28:11.458ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2122026-08-25T09:28:11.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2132026-08-25T09:28:11.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2142026-08-25T09:28:11.459ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2152026-08-25T09:28:11.459ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2162026-08-25T09:28:11.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2172026-08-25T09:28:11.460ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
218 Aug 25 09:28:20.144 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2447, tv_nsec: 837661730 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
219 Aug 25 09:28:20.145 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
220 Aug 25 09:28:20.146 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
2212026-08-25T09:28:21.458ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2222026-08-25T09:28:21.458ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2232026-08-25T09:28:21.459ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2242026-08-25T09:28:21.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2252026-08-25T09:28:21.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2262026-08-25T09:28:21.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2272026-08-25T09:28:21.461ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2282026-08-25T09:28:21.461ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
2292026-08-25T09:28:31.459ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2302026-08-25T09:28:31.459ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2312026-08-25T09:28:31.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2322026-08-25T09:28:31.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2332026-08-25T09:28:31.460ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2342026-08-25T09:28:31.461ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2352026-08-25T09:28:31.461ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2362026-08-25T09:28:31.461ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
237 Aug 25 09:28:35.147 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
238 Aug 25 09:28:35.147 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 143.375664355s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
2392026-08-25T09:28:41.459ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2402026-08-25T09:28:41.459ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2412026-08-25T09:28:41.460ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2422026-08-25T09:28:41.461ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2432026-08-25T09:28:41.461ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2442026-08-25T09:28:41.462ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2452026-08-25T09:28:41.462ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2462026-08-25T09:28:41.463ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
2472026-08-25T09:28:49.497ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::e4a1:b6ff:fee4:954a missed solicit interval
2482026-08-25T09:28:54.021ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2492026-08-25T09:28:54.021ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2502026-08-25T09:28:54.050ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2512026-08-25T09:28:54.068ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2522026-08-25T09:28:54.068ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2532026-08-25T09:28:54.068ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2542026-08-25T09:28:54.068ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2552026-08-25T09:28:54.069ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
256 Aug 25 09:29:00.294 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8eaca247-8fb9-4759-acd4-f15a549f2991.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("bb191099-ebdd-4b45-8c08-f1e2153936fe.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("d2bc0506-35d7-4fbf-857b-4f3f01b8596d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 2487, tv_nsec: 986597930 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
257 Aug 25 09:29:00.295 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
258 Aug 25 09:29:00.296 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
2592026-08-25T09:29:01.551ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2602026-08-25T09:29:01.551ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2612026-08-25T09:29:01.552ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2622026-08-25T09:29:01.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2632026-08-25T09:29:01.553ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2642026-08-25T09:29:01.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2652026-08-25T09:29:01.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2662026-08-25T09:29:01.553ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
2672026-08-25T09:29:11.551ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2682026-08-25T09:29:11.551ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2692026-08-25T09:29:11.551ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2702026-08-25T09:29:11.552ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2712026-08-25T09:29:11.552ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2722026-08-25T09:29:11.552ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2732026-08-25T09:29:11.552ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2742026-08-25T09:29:11.553ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
275 Aug 25 09:29:15.298 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
276 Aug 25 09:29:15.300 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 183.528192208s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
2772026-08-25T09:29:21.552ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2782026-08-25T09:29:21.552ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2792026-08-25T09:29:21.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2802026-08-25T09:29:21.554ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2812026-08-25T09:29:21.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2822026-08-25T09:29:21.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2832026-08-25T09:29:21.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2842026-08-25T09:29:21.554ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
2852026-08-25T09:29:31.551ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2862026-08-25T09:29:31.551ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2872026-08-25T09:29:31.552ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2882026-08-25T09:29:31.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2892026-08-25T09:29:31.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2902026-08-25T09:29:31.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2912026-08-25T09:29:31.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2922026-08-25T09:29:31.554ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
2932026-08-25T09:29:41.552ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2942026-08-25T09:29:41.552ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2952026-08-25T09:29:41.553ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2962026-08-25T09:29:41.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2972026-08-25T09:29:41.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2982026-08-25T09:29:41.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
2992026-08-25T09:29:41.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3002026-08-25T09:29:41.555ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
3012026-08-25T09:29:51.552ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
3022026-08-25T09:29:51.552ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
3032026-08-25T09:29:51.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3042026-08-25T09:29:51.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3052026-08-25T09:29:51.555ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3062026-08-25T09:29:51.555ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3072026-08-25T09:29:51.555ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3082026-08-25T09:29:51.555ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
3092026-08-25T09:30:01.552ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
3102026-08-25T09:30:01.552ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
3112026-08-25T09:30:01.553ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3122026-08-25T09:30:01.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3132026-08-25T09:30:01.554ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3142026-08-25T09:30:01.554ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3152026-08-25T09:30:01.555ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::e4a1:b6ff:fee4:954a on port Rear(Rear("rear0"))/0
3162026-08-25T09:30:01.555ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite