Max level shown:
1 [ Nov 8 10:27:09 Disabled. ]
2 [ Nov 8 10:27:09 Rereading configuration. ]
3 [ Nov 8 10:27:10 Enabled. ]
4 [ Nov 8 10:27:10 Rereading configuration. ]
5 [ Nov 8 10:27:10 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Nov 8 10:27:10 Method "start" exited with status 0. ]
72026-11-08T10:27:10.427ZINFOslog-rs: signal handler waiting for context
82026-11-08T10:27:10.449ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-11-08T10:27:10.449ZERROslog-rs: failed to start stats server: underlay address not found
102026-11-08T10:27:10.449ZINFOslog-rs: signal handler waiting got context
112026-11-08T10:27:10.451ZINFOslog-rs: admin: listening on [::]:8000
122026-11-08T10:27:11.449ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-11-08T10:27:12.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-11-08T10:27:13.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-11-08T10:27:14.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-11-08T10:27:15.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-11-08T10:27:16.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-11-08T10:27:17.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-11-08T10:27:18.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-11-08T10:27:19.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-11-08T10:27:20.450ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-11-08T10:27:21.450ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::ace7:99ff:fe71:f574 on if tfportrear0_0 index 3
232026-11-08T10:27:21.451ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
24 [ Nov 8 10:27:22 Rereading configuration. ]
25 [ Nov 8 10:27:22 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
26 + CONTRACT=''
27 + getopts c: o
28 + CONTRACT=213
29 + getopts c: o
30 + [[ -n 213 ]]
31 + pgrep -c 213
32 + PARENT_PID=2300
33 + [[ -n 2300 ]]
34 + pgrep -P 2300
35 + CHILD_PID=2302
36 + [[ -n 2302 ]]
37 + kill -USR1 2302
382026-11-08T10:27:22.257ZINFOslog-rs: handling smf refresh
392026-11-08T10:27:22.259ZINFOslog-rs: starting stats server on smf refresh
40 Nov 08 10:27:22.261 DEBG registered DTrace probes
41 [ Nov 8 10:27:22 Method "refresh" exited with status 0. ]
42 Nov 08 10:27:22.262 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
43 Nov 08 10:27:22.262 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
44 Nov 08 10:27:22.262 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
45 Nov 08 10:27:22.262 DEBG Nexus IP not provided, will use DNS to resolve it
46 Nov 08 10:27:22.263 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
47 Nov 08 10:27:22.264 DEBG starting producer registration task
48 Nov 08 10:27:22.264 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: dac62a7d-791f-432a-8ce1-900a5d3422f6, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
49 Nov 08 10:27:22.264 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
50 Nov 08 10:27:22.266 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41461, tv_nsec: 122837521 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
51 Nov 08 10:27:22.267 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
52 Nov 08 10:27:22.270 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
532026-11-08T10:27:22.974ZINFOslog-rs: peer change request current = {} requested = {} to_create = Difference(Iterate(Iter([]))) to_remove = Difference(Iterate(Iter([])))
542026-11-08T10:27:23.451ZINFOslog-rs: [tfportrear0_0] nbr is fe80::4022:b7ff:fe8a:3582@sock server
552026-11-08T10:27:23.451ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
562026-11-08T10:27:23.452ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::ace7:99ff:fe71:f574]:56797
572026-11-08T10:27:23.452ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
582026-11-08T10:27:23.454ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
592026-11-08T10:27:23.454ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
602026-11-08T10:27:23.455ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::4022:b7ff:fe8a:3582 on port Rear(Rear("rear0"))/0
612026-11-08T10:27:23.456ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::4022:b7ff:fe8a:3582 on port Rear(Rear("rear0"))/0
622026-11-08T10:27:23.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::4022:b7ff:fe8a:3582 on port Rear(Rear("rear0"))/0
632026-11-08T10:27:23.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::4022:b7ff:fe8a:3582 on port Rear(Rear("rear0"))/0
642026-11-08T10:27:23.457ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::4022:b7ff:fe8a:3582 on port Rear(Rear("rear0"))/0
652026-11-08T10:27:23.457ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
662026-11-08T10:27:23.458ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
672026-11-08T10:27:23.980ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd6d:2cc1:6dd7:81a7::1, vni: 99, metric: 18446744073709551615, }, }
68 Nov 08 10:27:37.449 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
69 Nov 08 10:27:37.449 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 15.008124496s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
70 Nov 08 10:27:37.789 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41476, tv_nsec: 469558618 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
71 Nov 08 10:27:37.790 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
72 Nov 08 10:27:37.790 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
732026-11-08T10:27:39.628ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
742026-11-08T10:27:41.628ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
75 Nov 08 10:27:52.751 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
76 Nov 08 10:27:52.751 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 30.352158158s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
77 Nov 08 10:27:53.097 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41491, tv_nsec: 819441961 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
78 Nov 08 10:27:53.098 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
79 Nov 08 10:27:53.098 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
802026-11-08T10:27:53.109ZINFOslog-rs: peer change request current = {} requested = {} to_create = Difference(Iterate(Iter([]))) to_remove = Difference(Iterate(Iter([])))
812026-11-08T10:28:05.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
822026-11-08T10:28:07.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
83 Nov 08 10:28:08.059 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
84 Nov 08 10:28:08.059 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 45.701405324s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
85 Nov 08 10:28:08.940 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41507, tv_nsec: 705183484 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
86 Nov 08 10:28:08.941 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
87 Nov 08 10:28:08.941 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
882026-11-08T10:28:09.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
892026-11-08T10:28:11.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
902026-11-08T10:28:13.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
912026-11-08T10:28:15.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
922026-11-08T10:28:17.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
932026-11-08T10:28:19.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
942026-11-08T10:28:21.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
952026-11-08T10:28:23.031ZINFOslog-rs: peer change request current = {} requested = {} to_create = Difference(Iterate(Iter([]))) to_remove = Difference(Iterate(Iter([])))
962026-11-08T10:28:23.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
97 Nov 08 10:28:23.901 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
98 Nov 08 10:28:23.902 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 61.587064246s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
99 Nov 08 10:28:25.471 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41524, tv_nsec: 281557678 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
100 Nov 08 10:28:25.472 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
101 Nov 08 10:28:25.473 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1022026-11-08T10:28:25.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1032026-11-08T10:28:27.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1042026-11-08T10:28:29.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1052026-11-08T10:28:31.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1062026-11-08T10:28:33.629ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1072026-11-08T10:28:35.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1082026-11-08T10:28:37.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1092026-11-08T10:28:39.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
110 Nov 08 10:28:40.434 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
111 Nov 08 10:28:40.434 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 78.16455472s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1122026-11-08T10:28:41.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1132026-11-08T10:28:43.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
114 Nov 08 10:28:43.934 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41542, tv_nsec: 793852988 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
115 Nov 08 10:28:43.935 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
116 Nov 08 10:28:43.935 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1172026-11-08T10:28:45.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1182026-11-08T10:28:47.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1192026-11-08T10:28:49.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1202026-11-08T10:28:51.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1212026-11-08T10:28:52.951ZINFOslog-rs: peer change request current = {} requested = {} to_create = Difference(Iterate(Iter([]))) to_remove = Difference(Iterate(Iter([])))
1222026-11-08T10:28:53.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1232026-11-08T10:28:55.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1242026-11-08T10:28:57.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
125 Nov 08 10:28:58.896 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
126 Nov 08 10:28:58.896 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 96.676578919s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1272026-11-08T10:28:59.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1282026-11-08T10:29:01.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1292026-11-08T10:29:03.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
130 Nov 08 10:29:03.976 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41562, tv_nsec: 890670435 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
131 Nov 08 10:29:03.977 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
132 Nov 08 10:29:03.978 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1332026-11-08T10:29:05.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1342026-11-08T10:29:07.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1352026-11-08T10:29:09.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1362026-11-08T10:29:11.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1372026-11-08T10:29:13.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1382026-11-08T10:29:15.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1392026-11-08T10:29:17.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
140 Nov 08 10:29:18.939 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
141 Nov 08 10:29:18.939 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 116.773612007s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1422026-11-08T10:29:19.630ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1432026-11-08T10:29:21.631ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1442026-11-08T10:29:22.873ZINFOslog-rs: peer change request current = {} requested = {} to_create = Difference(Iterate(Iter([]))) to_remove = Difference(Iterate(Iter([])))
1452026-11-08T10:29:23.631ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1462026-11-08T10:29:25.631ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1472026-11-08T10:29:27.631ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1482026-11-08T10:29:29.631ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1492026-11-08T10:29:31.632ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
150 Nov 08 10:29:32.417 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("066b912f-35f9-4a2a-9a9b-f8f67b74da6c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("291cdda7-c59b-4bbb-91e4-57cd2fc467ae.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("e399b87b-d5aa-4e17-83d2-15ff97eba4ab.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 41591, tv_nsec: 406639296 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
151 Nov 08 10:29:32.420 DEBG will register with Nexus at [fd00:1122:3344:101::b]:12221, component: producer-registration-task
152 Nov 08 10:29:32.427 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::b]:12221/metrics/producers, method: POST, component: producer-registration-task
1532026-11-08T10:29:35.631ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1542026-11-08T10:29:37.631ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1552026-11-08T10:29:39.632ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1562026-11-08T10:29:41.632ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1572026-11-08T10:29:43.633ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
1582026-11-08T10:29:45.634ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::4022:b7ff:fe8a:3582 missed solicit interval
159 Nov 08 10:29:47.388 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::b]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
160 Nov 08 10:29:47.388 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out, delay: 145.296542632s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1612026-11-08T10:29:52.814ZINFOslog-rs: peer change request current = {} requested = {} to_create = Difference(Iterate(Iter([]))) to_remove = Difference(Iterate(Iter([])))