Max level shown:
1 [ Aug 19 21:08:19 Disabled. ]
2 [ Aug 19 21:08:19 Rereading configuration. ]
3 [ Aug 19 21:08:20 Enabled. ]
4 [ Aug 19 21:08:20 Rereading configuration. ]
5 [ Aug 19 21:08:20 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Aug 19 21:08:20 Method "start" exited with status 0. ]
72026-08-19T21:08:20.490ZINFOslog-rs: signal handler waiting for context
82026-08-19T21:08:20.513ZERROslog-rs: failed to start stats server: underlay address not found
92026-08-19T21:08:20.513ZINFOslog-rs: signal handler waiting got context
102026-08-19T21:08:20.513ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
112026-08-19T21:08:20.514ZINFOslog-rs: admin: listening on [::1]:8000
122026-08-19T21:08:21.514ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-08-19T21:08:22.514ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-08-19T21:08:23.514ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-08-19T21:08:24.514ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-08-19T21:08:25.517ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-08-19T21:08:26.515ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-08-19T21:08:27.515ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-08-19T21:08:28.515ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-08-19T21:08:29.515ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-08-19T21:08:30.515ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-08-19T21:08:31.515ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::18f4:4eff:fe69:2c0f on if tfportrear0_0 index 3
232026-08-19T21:08:31.515ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
242026-08-19T21:08:31.516ZINFOslog-rs: [tfportrear0_0] nbr is fe80::9489:57ff:fe87:9d8@sock server
252026-08-19T21:08:31.516ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
262026-08-19T21:08:31.516ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::18f4:4eff:fe69:2c0f]:56797
272026-08-19T21:08:31.516ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
28 [ Aug 19 21:08:31 Rereading configuration. ]
29 [ Aug 19 21:08:31 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 214"). ]
30 + CONTRACT=''
31 + getopts c: o
32 + CONTRACT=214
33 + getopts c: o
34 + [[ -n 214 ]]
35 + pgrep -c 214
36 + PARENT_PID=2296
37 + [[ -n 2296 ]]
38 + pgrep -P 2296
39 + CHILD_PID=2297
40 + [[ -n 2297 ]]
41 + kill -USR1 2297
422026-08-19T21:08:31.994ZINFOslog-rs: handling smf refresh
432026-08-19T21:08:31.995ZINFOslog-rs: starting stats server on smf refresh
44 [ Aug 19 21:08:31 Method "refresh" exited with status 0. ]
45 Aug 19 21:08:31.996 DEBG registered DTrace probes
46 Aug 19 21:08:31.997 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
47 Aug 19 21:08:31.997 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
48 Aug 19 21:08:31.997 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
49 Aug 19 21:08:31.997 DEBG Nexus IP not provided, will use DNS to resolve it
50 Aug 19 21:08:31.997 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
51 Aug 19 21:08:31.997 DEBG starting producer registration task
52 Aug 19 21:08:31.998 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
53 Aug 19 21:08:31.998 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
54 Aug 19 21:08:31.999 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 1.27925ms, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
55 Aug 19 21:08:32.215 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 217.06659ms, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
56 Aug 19 21:08:32.566 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1085, tv_nsec: 36997711 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
57 Aug 19 21:08:32.566 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
58 Aug 19 21:08:32.568 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
592026-08-19T21:08:33.519ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
602026-08-19T21:08:33.520ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
612026-08-19T21:08:33.520ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
622026-08-19T21:08:33.521ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::9489:57ff:fe87:9d8 on port Rear(Rear("rear0"))/0
632026-08-19T21:08:33.522ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::9489:57ff:fe87:9d8 on port Rear(Rear("rear0"))/0
642026-08-19T21:08:33.523ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::9489:57ff:fe87:9d8 on port Rear(Rear("rear0"))/0
652026-08-19T21:08:33.523ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::9489:57ff:fe87:9d8 on port Rear(Rear("rear0"))/0
662026-08-19T21:08:33.523ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::9489:57ff:fe87:9d8 on port Rear(Rear("rear0"))/0
672026-08-19T21:08:33.524ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
682026-08-19T21:08:33.524ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
692026-08-19T21:08:34.318ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fda6:5398:2539:9fb6::1, vni: 99, metric: 18446744073709551615, }, }
70 Aug 19 21:08:47.608 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
71 Aug 19 21:08:47.608 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 15.571901332s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
72 Aug 19 21:08:49.039 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1101, tv_nsec: 476464426 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
73 Aug 19 21:08:49.040 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
74 Aug 19 21:08:49.041 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
752026-08-19T21:08:49.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
762026-08-19T21:08:51.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
772026-08-19T21:08:53.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
782026-08-19T21:08:55.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
792026-08-19T21:08:57.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
802026-08-19T21:08:59.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
812026-08-19T21:09:01.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
822026-08-19T21:09:03.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
83 Aug 19 21:09:04.000 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
84 Aug 19 21:09:04.001 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 32.009193696s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
85 Aug 19 21:09:05.494 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1117, tv_nsec: 975650082 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
86 Aug 19 21:09:05.494 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
87 Aug 19 21:09:05.495 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
882026-08-19T21:09:05.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
892026-08-19T21:09:07.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
902026-08-19T21:09:09.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
912026-08-19T21:09:11.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
922026-08-19T21:09:13.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
932026-08-19T21:09:15.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
942026-08-19T21:09:17.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
952026-08-19T21:09:19.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
96 Aug 19 21:09:20.456 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
97 Aug 19 21:09:20.457 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 48.51000435s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
982026-08-19T21:09:21.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
992026-08-19T21:09:23.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
100 Aug 19 21:09:25.477 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1138, tv_nsec: 13316826 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
101 Aug 19 21:09:25.478 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
102 Aug 19 21:09:25.479 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1032026-08-19T21:09:25.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1042026-08-19T21:09:27.555ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
105 Aug 19 21:09:40.440 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
106 Aug 19 21:09:40.440 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 68.547558611s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
107 Aug 19 21:09:49.309 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1161, tv_nsec: 909905390 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
108 Aug 19 21:09:49.309 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
109 Aug 19 21:09:49.310 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1102026-08-19T21:09:55.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1112026-08-19T21:09:57.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1122026-08-19T21:09:59.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1132026-08-19T21:10:01.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1142026-08-19T21:10:03.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
115 Aug 19 21:10:04.271 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
116 Aug 19 21:10:04.271 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 92.443625303s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1172026-08-19T21:10:05.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1182026-08-19T21:10:07.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1192026-08-19T21:10:09.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1202026-08-19T21:10:11.557ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1212026-08-19T21:10:13.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1222026-08-19T21:10:15.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1232026-08-19T21:10:17.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1242026-08-19T21:10:19.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1252026-08-19T21:10:21.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1262026-08-19T21:10:23.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
127 Aug 19 21:10:25.513 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1198, tv_nsec: 212304622 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
128 Aug 19 21:10:25.513 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
129 Aug 19 21:10:25.514 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1302026-08-19T21:10:25.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1312026-08-19T21:10:27.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1322026-08-19T21:10:29.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1332026-08-19T21:10:31.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1342026-08-19T21:10:33.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1352026-08-19T21:10:35.558ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1362026-08-19T21:10:37.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1372026-08-19T21:10:39.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
138 Aug 19 21:10:40.474 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
139 Aug 19 21:10:40.474 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 128.745369119s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1402026-08-19T21:10:41.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1412026-08-19T21:10:43.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1422026-08-19T21:10:45.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1432026-08-19T21:10:47.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1442026-08-19T21:10:49.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1452026-08-19T21:10:51.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1462026-08-19T21:10:53.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1472026-08-19T21:10:55.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1482026-08-19T21:10:57.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1492026-08-19T21:10:59.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1502026-08-19T21:11:01.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
151 Aug 19 21:11:02.834 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1235, tv_nsec: 635679430 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
152 Aug 19 21:11:02.835 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
153 Aug 19 21:11:02.836 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1542026-08-19T21:11:03.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1552026-08-19T21:11:05.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1562026-08-19T21:11:07.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1572026-08-19T21:11:09.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1582026-08-19T21:11:11.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1592026-08-19T21:11:13.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1602026-08-19T21:11:15.559ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1612026-08-19T21:11:17.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
162 Aug 19 21:11:17.797 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
163 Aug 19 21:11:17.797 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 166.16970405s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1642026-08-19T21:11:19.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1652026-08-19T21:11:21.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1662026-08-19T21:11:23.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1672026-08-19T21:11:25.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1682026-08-19T21:11:27.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1692026-08-19T21:11:29.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1702026-08-19T21:11:31.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
1712026-08-19T21:11:33.560ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
172 Aug 19 21:12:24.551 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1317, tv_nsec: 421300643 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
173 Aug 19 21:12:24.564 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
174 Aug 19 21:12:24.601 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1752026-08-19T21:12:39.591ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
176 Aug 19 21:12:39.604 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
177 Aug 19 21:12:39.608 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 248.02130994s, count: 9, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1782026-08-19T21:13:33.611ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::9489:57ff:fe87:9d8 missed solicit interval
179 Aug 19 21:14:47.407 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("3c52f70f-80c7-45af-8ad8-b3b54c9868ff.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("45dd640c-8534-406a-a0d6-3546b67214ca.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("569f1ceb-178a-439c-a40f-1ebe574c2c70.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1460, tv_nsec: 290461165 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
180 Aug 19 21:14:47.408 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
181 Aug 19 21:14:47.409 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
182 Aug 19 21:14:47.476 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "3efe92d5-3522-48ee-9431-3e0a3201cd45", "content-length": "41", "date": "Wed, 19 Aug 2026 21:14:47 GMT"} }), component: producer-registration-task
183 Aug 19 21:14:47.476 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
184 Aug 19 21:14:53.571 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057
185 Aug 19 21:14:53.577 INFO request completed, latency_us: 6292, response_code: 200, uri: /739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, method: GET, req_id: 7c30b130-17ec-4bd2-a6f7-6f5aa7831dec, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
186 Aug 19 21:14:54.571 INFO request completed, latency_us: 1001, response_code: 200, uri: /739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, method: GET, req_id: ec497893-d37d-4dda-b01b-fe582e80ddee, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
187 Aug 19 21:14:55.571 INFO request completed, latency_us: 684, response_code: 200, uri: /739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, method: GET, req_id: 5563dcaa-2b72-4d64-b520-cb3d4ef6997a, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
188 Aug 19 21:14:56.570 INFO request completed, latency_us: 705, response_code: 200, uri: /739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, method: GET, req_id: 5317f3ea-b892-4291-8ad5-42018ae7c59d, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
189 Aug 19 21:14:57.571 INFO request completed, latency_us: 663, response_code: 200, uri: /739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, method: GET, req_id: d3e9442e-e0e3-4b42-b531-06f359ce35fd, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
190 Aug 19 21:14:58.571 INFO request completed, latency_us: 645, response_code: 200, uri: /739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, method: GET, req_id: df7cf957-fffa-4d03-b7e7-0159dcb65c49, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
191 Aug 19 21:14:59.571 INFO request completed, latency_us: 630, response_code: 200, uri: /739ba3b4-9a57-41c7-8e4c-a707ad2ac51a, method: GET, req_id: 6d0b8ca0-e842-44cc-b546-188a09737a7f, remote_addr: [fd00:1122:3344:101::d]:57850, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874