Max level shown:
1 [ Aug 31 16:27:43 Disabled. ]
2 [ Aug 31 16:27:43 Rereading configuration. ]
3 [ Aug 31 16:27:44 Enabled. ]
4 [ Aug 31 16:27:44 Rereading configuration. ]
5 [ Aug 31 16:27:44 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Aug 31 16:27:44 Method "start" exited with status 0. ]
72026-08-31T16:27:44.785ZINFOslog-rs: signal handler waiting for context
82026-08-31T16:27:44.807ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
92026-08-31T16:27:44.807ZERROslog-rs: failed to start stats server: underlay address not found
102026-08-31T16:27:44.807ZINFOslog-rs: signal handler waiting got context
112026-08-31T16:27:44.809ZINFOslog-rs: admin: listening on [::]:8000
122026-08-31T16:27:45.807ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-08-31T16:27:46.807ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-08-31T16:27:47.807ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-08-31T16:27:48.807ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-08-31T16:27:49.807ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-08-31T16:27:50.807ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-08-31T16:27:51.808ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-08-31T16:27:52.808ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-08-31T16:27:53.808ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-08-31T16:27:54.808ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-08-31T16:27:55.808ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::d096:edff:fe57:65ab on if tfportrear0_0 index 3
232026-08-31T16:27:55.808ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
24 [ Aug 31 16:27:55 Rereading configuration. ]
25 [ Aug 31 16:27:55 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 212"). ]
26 + CONTRACT=''
27 + getopts c: o
28 + CONTRACT=212
29 + getopts c: o
30 + [[ -n 212 ]]
31 + pgrep -c 212
32 + PARENT_PID=2289
33 + [[ -n 2289 ]]
34 + pgrep -P 2289
35 + CHILD_PID=2293
36 + [[ -n 2293 ]]
37 + kill -USR1 2293
382026-08-31T16:27:55.943ZINFOslog-rs: handling smf refresh
39 [ Aug 31 16:27:55 Method "refresh" exited with status 0. ]
402026-08-31T16:27:55.944ZINFOslog-rs: starting stats server on smf refresh
41 Aug 31 16:27:55.945 DEBG registered DTrace probes
42 Aug 31 16:27:55.946 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
43 Aug 31 16:27:55.946 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
44 Aug 31 16:27:55.946 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
45 Aug 31 16:27:55.946 DEBG Nexus IP not provided, will use DNS to resolve it
46 Aug 31 16:27:55.946 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
47 Aug 31 16:27:55.946 DEBG starting producer registration task
48 Aug 31 16:27:55.946 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: b7c19bcd-9a6a-4d3c-8ca9-de4a2fcab9e2, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
49 Aug 31 16:27:55.946 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
50 Aug 31 16:27:55.948 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2de03509-a130-46f9-b202-200f0008608c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("97eb4bac-a1da-4d58-bc01-6d83ce489de8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 781, tv_nsec: 44109184 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
51 Aug 31 16:27:55.948 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
52 Aug 31 16:27:55.949 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
532026-08-31T16:28:04.813ZWARNslog-rs: could not list multicast groups to seed sweep, retrying later: Communication Error: error sending request for url (http://localhost:12224/multicast/groups)
542026-08-31T16:28:07.809ZINFOslog-rs: [tfportrear0_0] nbr is fe80::141c:4bff:fe9e:68d8@sock server
552026-08-31T16:28:07.809ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
562026-08-31T16:28:07.809ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::d096:edff:fe57:65ab]:56797
572026-08-31T16:28:07.810ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
582026-08-31T16:28:07.812ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
592026-08-31T16:28:07.812ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
602026-08-31T16:28:07.813ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
612026-08-31T16:28:07.814ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
622026-08-31T16:28:07.815ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
632026-08-31T16:28:07.815ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
642026-08-31T16:28:07.815ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
652026-08-31T16:28:07.816ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
662026-08-31T16:28:07.816ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
672026-08-31T16:28:09.993ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdc5:8c1d:5afb:16e7::1, vni: 99, metric: 18446744073709551615, }, }
68 Aug 31 16:28:10.951 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
69 Aug 31 16:28:10.951 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 15.004234331s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
70 Aug 31 16:28:11.103 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2de03509-a130-46f9-b202-200f0008608c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("97eb4bac-a1da-4d58-bc01-6d83ce489de8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 796, tv_nsec: 198996992 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
71 Aug 31 16:28:11.103 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
72 Aug 31 16:28:11.104 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
732026-08-31T16:28:18.157ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
742026-08-31T16:28:18.157ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
752026-08-31T16:28:18.157ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
762026-08-31T16:28:18.158ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
772026-08-31T16:28:18.159ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
782026-08-31T16:28:18.159ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
792026-08-31T16:28:18.159ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
802026-08-31T16:28:18.159ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
812026-08-31T16:28:18.160ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
822026-08-31T16:28:20.157ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
832026-08-31T16:28:22.157ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
842026-08-31T16:28:24.157ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
852026-08-31T16:28:26.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
86 Aug 31 16:28:26.426 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
87 Aug 31 16:28:26.426 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 30.159375615s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
88 Aug 31 16:28:26.983 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2de03509-a130-46f9-b202-200f0008608c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("97eb4bac-a1da-4d58-bc01-6d83ce489de8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 811, tv_nsec: 761702146 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
89 Aug 31 16:28:26.984 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
90 Aug 31 16:28:26.985 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
912026-08-31T16:28:28.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
922026-08-31T16:28:28.159ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
932026-08-31T16:28:28.159ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
942026-08-31T16:28:28.160ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
952026-08-31T16:28:28.160ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
962026-08-31T16:28:28.160ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
972026-08-31T16:28:28.161ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
982026-08-31T16:28:28.161ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
992026-08-31T16:28:28.161ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1002026-08-31T16:28:30.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1012026-08-31T16:28:32.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1022026-08-31T16:28:34.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1032026-08-31T16:28:36.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1042026-08-31T16:28:38.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1052026-08-31T16:28:38.159ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1062026-08-31T16:28:38.159ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1072026-08-31T16:28:38.160ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1082026-08-31T16:28:38.161ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1092026-08-31T16:28:38.161ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1102026-08-31T16:28:38.161ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1112026-08-31T16:28:38.161ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1122026-08-31T16:28:38.162ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1132026-08-31T16:28:40.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
114 Aug 31 16:28:41.945 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
115 Aug 31 16:28:41.945 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 45.721443183s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1162026-08-31T16:28:42.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
117 Aug 31 16:28:42.755 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2de03509-a130-46f9-b202-200f0008608c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("97eb4bac-a1da-4d58-bc01-6d83ce489de8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 827, tv_nsec: 576794742 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
118 Aug 31 16:28:42.756 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
119 Aug 31 16:28:42.756 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1202026-08-31T16:28:44.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1212026-08-31T16:28:46.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1222026-08-31T16:28:48.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1232026-08-31T16:28:48.160ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1242026-08-31T16:28:48.160ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1252026-08-31T16:28:48.161ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1262026-08-31T16:28:48.162ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1272026-08-31T16:28:48.162ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1282026-08-31T16:28:48.162ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1292026-08-31T16:28:48.162ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1302026-08-31T16:28:48.163ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1312026-08-31T16:28:50.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1322026-08-31T16:28:52.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1332026-08-31T16:28:54.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1342026-08-31T16:28:56.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
135 Aug 31 16:28:57.716 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
136 Aug 31 16:28:57.717 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 61.536192047s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1372026-08-31T16:28:58.158ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1382026-08-31T16:28:58.161ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1392026-08-31T16:28:58.161ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1402026-08-31T16:28:58.162ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1412026-08-31T16:28:58.163ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1422026-08-31T16:28:58.163ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1432026-08-31T16:28:58.163ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1442026-08-31T16:28:58.163ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1452026-08-31T16:28:58.163ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
146 Aug 31 16:28:58.752 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2de03509-a130-46f9-b202-200f0008608c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("97eb4bac-a1da-4d58-bc01-6d83ce489de8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 843, tv_nsec: 616971023 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
147 Aug 31 16:28:58.752 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
148 Aug 31 16:28:58.753 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1492026-08-31T16:29:00.159ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1502026-08-31T16:29:02.160ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1512026-08-31T16:29:04.161ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1522026-08-31T16:29:06.162ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1532026-08-31T16:29:08.162ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1542026-08-31T16:29:08.162ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1552026-08-31T16:29:08.163ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1562026-08-31T16:29:08.164ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1572026-08-31T16:29:08.164ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1582026-08-31T16:29:08.165ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1592026-08-31T16:29:08.165ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1602026-08-31T16:29:08.165ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1612026-08-31T16:29:12.164ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
162 Aug 31 16:29:13.718 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
163 Aug 31 16:29:13.719 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 77.576650425s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1642026-08-31T16:29:18.163ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1652026-08-31T16:29:18.163ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1662026-08-31T16:29:18.164ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1672026-08-31T16:29:18.164ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1682026-08-31T16:29:18.165ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1692026-08-31T16:29:18.165ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1702026-08-31T16:29:18.165ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1712026-08-31T16:29:18.165ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
172 Aug 31 16:29:19.284 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2de03509-a130-46f9-b202-200f0008608c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("97eb4bac-a1da-4d58-bc01-6d83ce489de8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 864, tv_nsec: 183998581 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
173 Aug 31 16:29:19.285 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
174 Aug 31 16:29:19.286 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1752026-08-31T16:29:28.163ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1762026-08-31T16:29:28.163ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1772026-08-31T16:29:28.165ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1782026-08-31T16:29:28.165ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1792026-08-31T16:29:28.166ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1802026-08-31T16:29:28.166ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1812026-08-31T16:29:28.166ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1822026-08-31T16:29:28.166ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
183 Aug 31 16:29:34.289 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
184 Aug 31 16:29:34.289 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 98.143892143s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1852026-08-31T16:29:40.088ZWARNslog-rs: [tfportrear0_0] neighbor sock@fe80::141c:4bff:fe9e:68d8 missed solicit interval
1862026-08-31T16:29:40.315ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1872026-08-31T16:29:40.315ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1882026-08-31T16:29:40.339ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1892026-08-31T16:29:40.361ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1902026-08-31T16:29:40.361ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1912026-08-31T16:29:40.362ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1922026-08-31T16:29:40.362ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
1932026-08-31T16:29:40.362ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
194 Aug 31 16:29:40.611 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("2de03509-a130-46f9-b202-200f0008608c.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("97eb4bac-a1da-4d58-bc01-6d83ce489de8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("ed39db1d-f9a4-4481-98a4-d08a7faf8f62.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 885, tv_nsec: 482763420 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
195 Aug 31 16:29:40.615 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
196 Aug 31 16:29:40.627 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1972026-08-31T16:29:49.692ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1982026-08-31T16:29:49.693ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1992026-08-31T16:29:49.753ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2002026-08-31T16:29:49.776ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2012026-08-31T16:29:49.778ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2022026-08-31T16:29:49.778ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2032026-08-31T16:29:49.780ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2042026-08-31T16:29:49.783ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
205 Aug 31 16:29:55.645 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
206 Aug 31 16:29:55.650 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 119.501018387s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
2072026-08-31T16:29:59.616ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
2082026-08-31T16:29:59.617ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
2092026-08-31T16:29:59.618ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2102026-08-31T16:29:59.619ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2112026-08-31T16:29:59.619ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2122026-08-31T16:29:59.619ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2132026-08-31T16:29:59.620ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:a02a:: -> fe80::141c:4bff:fe9e:68d8 on port Rear(Rear("rear0"))/0
2142026-08-31T16:29:59.620ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite