Max level shown:
1 [ Nov 6 00:22:53 Disabled. ]
2 [ Nov 6 00:22:53 Rereading configuration. ]
3 [ Nov 6 00:22:53 Enabled. ]
4 [ Nov 6 00:22:53 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
5 [ Nov 6 00:22:53 Method "start" exited with status 0. ]
62026-11-06T00:22:53.159ZINFOslog-rs: signal handler waiting for context
72026-11-06T00:22:53.175ZINFOslog-rs: signal handler waiting got context
82026-11-06T00:22:53.176ZINFOslog-rs: [net1] sm initialized with addr fe80::6c8d:a9ff:fe83:9fd7 on if net1 index 4
92026-11-06T00:22:53.176ZINFOslog-rs: [net0] sm initialized with addr fe80::e857:59ff:fe66:facc on if net0 index 3
102026-11-06T00:22:53.176ZDEBGslog-rs: [net1] starting discovery handler
112026-11-06T00:22:53.176ZDEBGslog-rs: [net0] starting discovery handler
122026-11-06T00:22:53.176ZINFOslog-rs: admin: listening on [::1]:8000
132026-11-06T00:22:53.616ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }}))
142026-11-06T00:22:53.616ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }}))
152026-11-06T00:22:56.545ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }}))
162026-11-06T00:22:56.545ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }}))
172026-11-06T00:23:09.625ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }}))
182026-11-06T00:23:09.625ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }}))
192026-11-06T00:23:09.625ZWARNslog-rs: [net1] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }, Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }}))
202026-11-06T00:23:09.625ZWARNslog-rs: [net0] admin event in solicit state: Announce(Underlay({Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }, Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }}))
212026-11-06T00:23:17.177ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
222026-11-06T00:23:17.177ZDEBGslog-rs: [net0] transition solicit -> exchange
232026-11-06T00:23:17.177ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
242026-11-06T00:23:17.178ZINFOslog-rs: [net0] waiting for exchange server to start
252026-11-06T00:23:19.430ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
262026-11-06T00:23:21.681ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
272026-11-06T00:23:23.932ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
282026-11-06T00:23:26.184ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
292026-11-06T00:23:28.435ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
302026-11-06T00:23:29.177ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
312026-11-06T00:23:30.686ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
322026-11-06T00:23:31.177ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
332026-11-06T00:23:31.177ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
342026-11-06T00:23:32.938ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
352026-11-06T00:23:32.938ZINFOslog-rs: [net0] removing 0 routes from illumos
36 Nov 06 00:23:53.615 DEBG registered DTrace probes
37 Nov 06 00:23:53.615 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot
38 Nov 06 00:23:53.615 INFO listening, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
39 Nov 06 00:23:53.615 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::1]:8001, component: dropshot
40 Nov 06 00:23:53.615 DEBG Nexus IP not provided, will use DNS to resolve it
41 Nov 06 00:23:53.615 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
42 Nov 06 00:23:53.615 DEBG starting producer registration task
43 Nov 06 00:23:53.615 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::1]:8001, producer_id: 77b206d1-b803-4af9-823e-c12c10a45114, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
44 Nov 06 00:23:53.615 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
45 Nov 06 00:23:53.616 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 637, tv_nsec: 691428056 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
46 Nov 06 00:23:53.617 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
47 Nov 06 00:23:53.618 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
48 Nov 06 00:24:08.619 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
49 Nov 06 00:24:08.619 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 15.004485938s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
50 Nov 06 00:24:08.951 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 653, tv_nsec: 26852042 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
51 Nov 06 00:24:08.952 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
52 Nov 06 00:24:08.953 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
532026-11-06T00:24:15.179ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
542026-11-06T00:24:15.179ZDEBGslog-rs: [net0] transition solicit -> exchange
552026-11-06T00:24:15.179ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
562026-11-06T00:24:15.179ZINFOslog-rs: [net0] waiting for exchange server to start
572026-11-06T00:24:17.431ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
582026-11-06T00:24:19.683ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
592026-11-06T00:24:21.934ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
60 Nov 06 00:24:23.954 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
61 Nov 06 00:24:23.954 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 30.339406856s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
622026-11-06T00:24:24.185ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
63 Nov 06 00:24:24.259 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 668, tv_nsec: 334620557 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
64 Nov 06 00:24:24.260 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
65 Nov 06 00:24:24.261 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
662026-11-06T00:24:26.437ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
672026-11-06T00:24:27.178ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
682026-11-06T00:24:28.688ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
692026-11-06T00:24:29.178ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
702026-11-06T00:24:29.178ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
712026-11-06T00:24:30.940ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
722026-11-06T00:24:30.940ZINFOslog-rs: [net0] removing 0 routes from illumos
73 Nov 06 00:24:39.261 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
74 Nov 06 00:24:39.261 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 45.646094484s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
75 Nov 06 00:24:40.202 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 684, tv_nsec: 277451200 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
76 Nov 06 00:24:40.203 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
77 Nov 06 00:24:40.204 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
78 Nov 06 00:24:55.204 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
79 Nov 06 00:24:55.204 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 61.589035762s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
80 Nov 06 00:24:57.776 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 701, tv_nsec: 851376729 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
81 Nov 06 00:24:57.777 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
82 Nov 06 00:24:57.777 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
83 Nov 06 00:25:12.778 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
84 Nov 06 00:25:12.778 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 79.163224268s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
852026-11-06T00:25:13.180ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
862026-11-06T00:25:13.180ZDEBGslog-rs: [net0] transition solicit -> exchange
872026-11-06T00:25:13.181ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
882026-11-06T00:25:13.181ZINFOslog-rs: [net0] waiting for exchange server to start
892026-11-06T00:25:15.433ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
90 Nov 06 00:25:16.652 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 720, tv_nsec: 727543007 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
91 Nov 06 00:25:16.653 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
92 Nov 06 00:25:16.653 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
932026-11-06T00:25:17.684ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
942026-11-06T00:25:19.936ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
952026-11-06T00:25:22.187ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
962026-11-06T00:25:24.438ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
972026-11-06T00:25:25.179ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
982026-11-06T00:25:26.690ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
992026-11-06T00:25:27.179ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
1002026-11-06T00:25:27.179ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
1012026-11-06T00:25:28.941ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1022026-11-06T00:25:28.941ZINFOslog-rs: [net0] removing 0 routes from illumos
103 Nov 06 00:25:31.655 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
104 Nov 06 00:25:31.655 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 98.03965189s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
105 Nov 06 00:25:36.965 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 741, tv_nsec: 40382962 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
106 Nov 06 00:25:36.966 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
107 Nov 06 00:25:36.966 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
108 Nov 06 00:25:51.967 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
109 Nov 06 00:25:51.967 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 118.351945483s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
110 Nov 06 00:26:07.326 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 771, tv_nsec: 401692077 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
111 Nov 06 00:26:07.327 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
112 Nov 06 00:26:07.328 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1132026-11-06T00:26:11.182ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
1142026-11-06T00:26:11.182ZDEBGslog-rs: [net0] transition solicit -> exchange
1152026-11-06T00:26:11.182ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
1162026-11-06T00:26:11.182ZINFOslog-rs: [net0] waiting for exchange server to start
1172026-11-06T00:26:13.434ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1182026-11-06T00:26:15.685ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1192026-11-06T00:26:17.937ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1202026-11-06T00:26:20.188ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
121 Nov 06 00:26:22.328 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
122 Nov 06 00:26:22.328 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 148.713027435s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1232026-11-06T00:26:22.439ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1242026-11-06T00:26:23.180ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
1252026-11-06T00:26:24.691ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1262026-11-06T00:26:25.180ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
1272026-11-06T00:26:25.180ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
1282026-11-06T00:26:26.941ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1292026-11-06T00:26:26.941ZINFOslog-rs: [net0] removing 0 routes from illumos
130 Nov 06 00:26:47.995 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 812, tv_nsec: 70679084 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
131 Nov 06 00:26:47.996 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
132 Nov 06 00:26:47.997 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
133 Nov 06 00:27:02.998 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
134 Nov 06 00:27:02.999 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 189.383503693s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1352026-11-06T00:27:09.183ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
1362026-11-06T00:27:09.183ZDEBGslog-rs: [net0] transition solicit -> exchange
1372026-11-06T00:27:09.184ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
1382026-11-06T00:27:09.184ZINFOslog-rs: [net0] waiting for exchange server to start
1392026-11-06T00:27:11.436ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1402026-11-06T00:27:13.688ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1412026-11-06T00:27:15.939ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1422026-11-06T00:27:18.191ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1432026-11-06T00:27:20.442ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1442026-11-06T00:27:21.182ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
1452026-11-06T00:27:22.693ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1462026-11-06T00:27:23.182ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
1472026-11-06T00:27:23.182ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
1482026-11-06T00:27:24.945ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1492026-11-06T00:27:24.945ZINFOslog-rs: [net0] removing 0 routes from illumos
150 Nov 06 00:27:36.720 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 860, tv_nsec: 795722263 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
151 Nov 06 00:27:36.721 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
152 Nov 06 00:27:36.722 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
153 Nov 06 00:27:51.723 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
154 Nov 06 00:27:51.723 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 238.108164422s, count: 9, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1552026-11-06T00:27:57.414ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
1562026-11-06T00:27:57.414ZDEBGslog-rs: [net0] transition solicit -> exchange
1572026-11-06T00:27:57.414ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
1582026-11-06T00:27:57.414ZINFOslog-rs: [net0] waiting for exchange server to start
1592026-11-06T00:27:59.666ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1602026-11-06T00:28:01.918ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1612026-11-06T00:28:04.168ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1622026-11-06T00:28:06.419ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1632026-11-06T00:28:08.670ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1642026-11-06T00:28:10.921ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1652026-11-06T00:28:13.172ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1662026-11-06T00:28:15.423ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1672026-11-06T00:28:17.674ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1682026-11-06T00:28:19.183ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
1692026-11-06T00:28:19.925ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1702026-11-06T00:28:21.183ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
1712026-11-06T00:28:21.183ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
1722026-11-06T00:28:22.177ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1732026-11-06T00:28:22.177ZINFOslog-rs: [net0] removing 0 routes from illumos
1742026-11-06T00:29:13.187ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
1752026-11-06T00:29:13.187ZDEBGslog-rs: [net0] transition solicit -> exchange
1762026-11-06T00:29:13.187ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
1772026-11-06T00:29:13.187ZINFOslog-rs: [net0] waiting for exchange server to start
178 Nov 06 00:29:13.406 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 957, tv_nsec: 480970806 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
179 Nov 06 00:29:13.406 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
180 Nov 06 00:29:13.407 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1812026-11-06T00:29:15.438ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1822026-11-06T00:29:17.690ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1832026-11-06T00:29:19.941ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1842026-11-06T00:29:22.192ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1852026-11-06T00:29:24.444ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1862026-11-06T00:29:25.184ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
1872026-11-06T00:29:26.695ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1882026-11-06T00:29:27.184ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
1892026-11-06T00:29:27.184ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
190 Nov 06 00:29:28.408 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
191 Nov 06 00:29:28.408 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 334.793391336s, count: 10, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1922026-11-06T00:29:28.946ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1932026-11-06T00:29:28.946ZINFOslog-rs: [net0] removing 0 routes from illumos
1942026-11-06T00:30:11.188ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
1952026-11-06T00:30:11.188ZDEBGslog-rs: [net0] transition solicit -> exchange
1962026-11-06T00:30:11.188ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
1972026-11-06T00:30:11.188ZINFOslog-rs: [net0] waiting for exchange server to start
1982026-11-06T00:30:13.440ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
1992026-11-06T00:30:15.692ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2002026-11-06T00:30:17.943ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2012026-11-06T00:30:20.194ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2022026-11-06T00:30:22.446ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2032026-11-06T00:30:23.185ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
2042026-11-06T00:30:24.697ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2052026-11-06T00:30:25.185ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
2062026-11-06T00:30:25.185ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
2072026-11-06T00:30:26.948ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2082026-11-06T00:30:26.948ZINFOslog-rs: [net0] removing 0 routes from illumos
2092026-11-06T00:31:09.190ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
2102026-11-06T00:31:09.190ZDEBGslog-rs: [net0] transition solicit -> exchange
2112026-11-06T00:31:09.190ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
2122026-11-06T00:31:09.190ZINFOslog-rs: [net0] waiting for exchange server to start
2132026-11-06T00:31:11.442ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2142026-11-06T00:31:13.693ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2152026-11-06T00:31:15.944ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2162026-11-06T00:31:18.196ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2172026-11-06T00:31:20.447ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2182026-11-06T00:31:21.187ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
2192026-11-06T00:31:22.698ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2202026-11-06T00:31:23.187ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
2212026-11-06T00:31:23.187ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
2222026-11-06T00:31:24.949ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2232026-11-06T00:31:24.950ZINFOslog-rs: [net0] removing 0 routes from illumos
2242026-11-06T00:32:07.191ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
2252026-11-06T00:32:07.191ZDEBGslog-rs: [net0] transition solicit -> exchange
2262026-11-06T00:32:07.191ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
2272026-11-06T00:32:07.191ZINFOslog-rs: [net0] waiting for exchange server to start
2282026-11-06T00:32:09.443ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2292026-11-06T00:32:11.695ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2302026-11-06T00:32:13.946ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2312026-11-06T00:32:16.197ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2322026-11-06T00:32:18.449ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2332026-11-06T00:32:19.188ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
2342026-11-06T00:32:20.699ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2352026-11-06T00:32:21.188ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
2362026-11-06T00:32:21.188ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
2372026-11-06T00:32:22.950ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2382026-11-06T00:32:22.950ZINFOslog-rs: [net0] removing 0 routes from illumos
2392026-11-06T00:32:57.414ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
2402026-11-06T00:32:57.414ZDEBGslog-rs: [net0] transition solicit -> exchange
2412026-11-06T00:32:57.414ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
2422026-11-06T00:32:57.414ZINFOslog-rs: [net0] waiting for exchange server to start
2432026-11-06T00:32:59.667ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2442026-11-06T00:33:01.918ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2452026-11-06T00:33:04.169ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2462026-11-06T00:33:06.421ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2472026-11-06T00:33:08.672ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2482026-11-06T00:33:09.189ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
2492026-11-06T00:33:10.924ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2502026-11-06T00:33:11.189ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
2512026-11-06T00:33:11.189ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
2522026-11-06T00:33:13.174ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2532026-11-06T00:33:13.174ZINFOslog-rs: [net0] removing 0 routes from illumos
2542026-11-06T00:33:15.193ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
2552026-11-06T00:33:15.193ZDEBGslog-rs: [net0] transition solicit -> exchange
2562026-11-06T00:33:15.193ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
2572026-11-06T00:33:15.193ZINFOslog-rs: [net0] waiting for exchange server to start
2582026-11-06T00:33:17.445ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2592026-11-06T00:33:19.696ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2602026-11-06T00:33:21.948ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2612026-11-06T00:33:24.198ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2622026-11-06T00:33:26.449ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2632026-11-06T00:33:27.189ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 missed solicit interval
2642026-11-06T00:33:28.700ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2652026-11-06T00:33:29.189ZWARNslog-rs: [net0] neighbor swsock@fe80::7083:ddff:feee:b129 expire
2662026-11-06T00:33:29.189ZWARNslog-rs: [net0] expiring peer fe80::7083:ddff:feee:b129 due to discovery event
2672026-11-06T00:33:30.951ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2682026-11-06T00:33:30.952ZINFOslog-rs: [net0] removing 0 routes from illumos
269 Nov 06 00:33:52.006 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("5223e370-19c9-4db5-8642-6562160fa3f5.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("69025b2f-8c41-4e16-99c3-bebb9b66faf8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("8fb2997d-477a-4332-84ac-fcb89a3693f0.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::a)) }], valid_until: Instant { tv_sec: 1236, tv_nsec: 81681094 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
270 Nov 06 00:33:52.007 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
271 Nov 06 00:33:52.008 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
272 Nov 06 00:34:07.009 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
273 Nov 06 00:34:07.009 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 613.394457752s, count: 11, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
2742026-11-06T00:34:13.194ZINFOslog-rs: [net0] nbr is fe80::7083:ddff:feee:b129@swsock transit
2752026-11-06T00:34:13.194ZDEBGslog-rs: [net0] transition solicit -> exchange
2762026-11-06T00:34:13.194ZINFOslog-rs: [net0] exchange: listening on [fe80::e857:59ff:fe66:facc]:56797
2772026-11-06T00:34:13.194ZINFOslog-rs: [net0] waiting for exchange server to start
2782026-11-06T00:34:15.446ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2792026-11-06T00:34:17.698ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2802026-11-06T00:34:19.948ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2812026-11-06T00:34:22.199ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed
2822026-11-06T00:34:24.451ZWARNslog-rs: [net0] exchange pull: timeout error: deadline has elapsed