Max level shown:
1 [ Jul 16 13:11:00 Disabled. ]
2 [ Jul 16 13:11:00 Rereading configuration. ]
3 [ Jul 16 13:11:00 Enabled. ]
4 [ Jul 16 13:11:00 Rereading configuration. ]
5 [ Jul 16 13:11:01 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Jul 16 13:11:01 Method "start" exited with status 0. ]
72026-07-16T13:11:01.091ZINFOslog-rs: signal handler waiting for context
82026-07-16T13:11:01.113ZERROslog-rs: failed to start stats server: underlay address not found
92026-07-16T13:11:01.113ZINFOslog-rs: signal handler waiting got context
102026-07-16T13:11:01.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
112026-07-16T13:11:01.120ZINFOslog-rs: admin: listening on [::1]:8000
122026-07-16T13:11:02.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-07-16T13:11:03.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-07-16T13:11:04.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-07-16T13:11:05.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-07-16T13:11:06.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-07-16T13:11:07.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-07-16T13:11:08.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-07-16T13:11:09.113ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-07-16T13:11:10.114ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-07-16T13:11:11.114ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-07-16T13:11:12.114ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::fc51:6cff:fe8d:d230 on if tfportrear0_0 index 3
232026-07-16T13:11:12.114ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
242026-07-16T13:11:12.114ZINFOslog-rs: [tfportrear0_0] nbr is fe80::748f:6eff:fec8:348c@buskin server
252026-07-16T13:11:12.114ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
262026-07-16T13:11:12.115ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::fc51:6cff:fe8d:d230]:56797
272026-07-16T13:11:12.115ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
28 [ Jul 16 13:11:12 Rereading configuration. ]
29 [ Jul 16 13:11:12 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
30 + CONTRACT=''
31 + getopts c: o
32 + CONTRACT=213
33 + getopts c: o
34 + [[ -n 213 ]]
35 + pgrep -c 213
36 + PARENT_PID=2294
37 + [[ -n 2294 ]]
38 + pgrep -P 2294
39 + CHILD_PID=2295
40 + [[ -n 2295 ]]
41 + kill -USR1 2295
422026-07-16T13:11:12.675ZINFOslog-rs: handling smf refresh
43 [ Jul 16 13:11:12 Method "refresh" exited with status 0. ]
442026-07-16T13:11:12.678ZINFOslog-rs: starting stats server on smf refresh
45 Jul 16 13:11:12.679 DEBG registered DTrace probes
46 Jul 16 13:11:12.679 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
47 Jul 16 13:11:12.680 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
48 Jul 16 13:11:12.680 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
49 Jul 16 13:11:12.680 DEBG Nexus IP not provided, will use DNS to resolve it
50 Jul 16 13:11:12.680 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
51 Jul 16 13:11:12.680 DEBG starting producer registration task
52 Jul 16 13:11:12.680 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 18823c2d-71d6-4785-a63c-2b91da01f424, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
53 Jul 16 13:11:12.680 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
542026-07-16T13:11:13.888ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fd0a:fefc:9641:6ebf::1, vni: 99, metric: 18446744073709551615, }, }
552026-07-16T13:11:14.117ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
562026-07-16T13:11:14.118ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
572026-07-16T13:11:14.118ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
582026-07-16T13:11:14.120ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
592026-07-16T13:11:14.122ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
602026-07-16T13:11:14.122ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
612026-07-16T13:11:14.122ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
622026-07-16T13:11:14.123ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
632026-07-16T13:11:14.123ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
642026-07-16T13:11:14.124ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
65 Jul 16 13:11:17.683 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 701, tv_nsec: 868485213 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
66 Jul 16 13:11:17.684 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
67 Jul 16 13:11:17.685 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
682026-07-16T13:11:31.655ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
69 Jul 16 13:11:32.224 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
70 Jul 16 13:11:32.224 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 20.005756717s, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
71 Jul 16 13:11:32.589 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 717, tv_nsec: 238086305 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
72 Jul 16 13:11:32.590 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
73 Jul 16 13:11:32.591 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
742026-07-16T13:11:33.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
752026-07-16T13:11:35.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
762026-07-16T13:11:37.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
772026-07-16T13:11:39.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
782026-07-16T13:11:41.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
792026-07-16T13:11:43.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
802026-07-16T13:11:45.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
81 Jul 16 13:11:47.550 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
82 Jul 16 13:11:47.550 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 35.374751066s, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
832026-07-16T13:11:47.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
84 Jul 16 13:11:47.805 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 732, tv_nsec: 495523159 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
85 Jul 16 13:11:47.805 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
86 Jul 16 13:11:47.807 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
872026-07-16T13:11:49.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
882026-07-16T13:11:51.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
892026-07-16T13:11:53.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
902026-07-16T13:11:55.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
912026-07-16T13:11:57.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
922026-07-16T13:11:59.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
932026-07-16T13:12:01.656ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
94 Jul 16 13:12:02.767 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
95 Jul 16 13:12:02.767 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 50.632620309s, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
96 Jul 16 13:12:03.488 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 748, tv_nsec: 220720023 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
97 Jul 16 13:12:03.488 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
98 Jul 16 13:12:03.489 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
992026-07-16T13:12:03.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1002026-07-16T13:12:05.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1012026-07-16T13:12:07.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1022026-07-16T13:12:09.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1032026-07-16T13:12:11.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1042026-07-16T13:12:15.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1052026-07-16T13:12:17.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
106 Jul 16 13:12:18.450 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
107 Jul 16 13:12:18.450 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 66.358054768s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
108 Jul 16 13:12:21.438 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 766, tv_nsec: 219618029 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
109 Jul 16 13:12:21.438 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
110 Jul 16 13:12:21.439 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1112026-07-16T13:12:21.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1122026-07-16T13:12:23.657ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
113 Jul 16 13:12:36.400 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
114 Jul 16 13:12:36.400 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 84.356250309s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
115 Jul 16 13:12:39.652 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 784, tv_nsec: 482531577 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
116 Jul 16 13:12:39.652 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
117 Jul 16 13:12:39.653 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1182026-07-16T13:12:45.665ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
1192026-07-16T13:12:49.666ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c missed solicit interval
120 Jul 16 13:12:54.630 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
121 Jul 16 13:12:54.630 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 102.619406043s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
122 Jul 16 13:13:04.147 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 808, tv_nsec: 994029940 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
123 Jul 16 13:13:04.147 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
124 Jul 16 13:13:04.148 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
1252026-07-16T13:13:21.127ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::748f:6eff:fec8:348c expire
1262026-07-16T13:13:21.131ZWARNslog-rs: [tfportrear0_0] expiring peer fe80::748f:6eff:fec8:348c due to discovery event
1272026-07-16T13:13:21.131ZINFOslog-rs: [tfportrear0_0] removing routes 5 from dendrite
128 Jul 16 13:13:21.242 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
129 Jul 16 13:13:21.253 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 129.232291337s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
1302026-07-16T13:13:21.257ZINFOslog-rs: [tfportrear0_0] nbr is fe80::748f:6eff:fec8:348c@buskin server
1312026-07-16T13:13:22.108ZERROslog-rs: [tfportrear0_0] failed to remove tunnel routes: {} OPTE driver is not attached
1322026-07-16T13:13:22.108ZDEBGslog-rs: [tfportrear0_0] redistributing expire to 0 peers
1332026-07-16T13:13:22.109ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
1342026-07-16T13:13:22.109ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::fc51:6cff:fe8d:d230]:56797
1352026-07-16T13:13:22.109ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
1362026-07-16T13:13:22.615ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
1372026-07-16T13:13:22.616ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
1382026-07-16T13:13:22.755ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
1392026-07-16T13:13:22.757ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
1402026-07-16T13:13:22.757ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
1412026-07-16T13:13:22.757ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
1422026-07-16T13:13:22.758ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::748f:6eff:fec8:348c on port Rear(Rear("rear0"))/0
1432026-07-16T13:13:22.758ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
1442026-07-16T13:13:22.769ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
145 Jul 16 13:13:36.250 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 841, tv_nsec: 95755388 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
146 Jul 16 13:13:36.250 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
147 Jul 16 13:13:36.252 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
148 Jul 16 13:13:51.253 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
149 Jul 16 13:13:51.253 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out, delay: 159.234367419s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
150 Jul 16 13:14:29.213 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("67353701-791f-4a29-b3f9-4b182d603f45.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("a3a11592-17cc-424f-af85-24310e033bfb.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("fbbaab46-5bd5-45b8-8320-98b6b242ec6d.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::b)) }], valid_until: Instant { tv_sec: 894, tv_nsec: 60138168 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
151 Jul 16 13:14:29.214 DEBG will register with Nexus at [fd00:1122:3344:101::c]:12221, component: producer-registration-task
152 Jul 16 13:14:29.215 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::c]:12221/metrics/producers, method: POST, component: producer-registration-task
153 Jul 16 13:14:29.284 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::c]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "f0416fb8-4980-445d-a29f-f2a2df5873cd", "content-length": "41", "date": "Thu, 16 Jul 2026 13:14:29 GMT"} }), component: producer-registration-task
154 Jul 16 13:14:29.285 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
155 Jul 16 13:14:44.802 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057
156 Jul 16 13:14:44.808 INFO request completed, latency_us: 4744, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: f2bd026f-3858-425b-ac38-7f938d2aee73, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
157 Jul 16 13:14:45.803 INFO request completed, latency_us: 602, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: f5d3373f-b2a8-453b-97d1-e370c6424e88, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
158 Jul 16 13:14:46.802 INFO request completed, latency_us: 590, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: e9dc7068-c872-4632-a502-788aad4af4d4, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
159 Jul 16 13:14:47.803 INFO request completed, latency_us: 577, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: bda62406-af02-475b-ab15-241c92fa476b, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
160 Jul 16 13:14:48.802 INFO request completed, latency_us: 598, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: e352966f-a313-41c6-85dd-56ee7a647b1d, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
161 Jul 16 13:14:49.803 INFO request completed, latency_us: 584, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: dd1a56cf-5e67-402c-a7a5-b236e1818fe0, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
162 Jul 16 13:14:50.803 INFO request completed, latency_us: 715, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: a0007bd1-1cec-4742-8c00-dc5efb47100a, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
163 Jul 16 13:14:51.803 INFO request completed, latency_us: 795, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: bf44bb69-f96c-40ca-b169-4da5ff64e011, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
164 Jul 16 13:14:52.803 INFO request completed, latency_us: 674, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: b7cda5c8-de92-46d5-abfc-da2677fe4803, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
165 Jul 16 13:14:53.803 INFO request completed, latency_us: 835, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: 1ca29f3a-be30-4d09-82c8-9f1e2bde4dcc, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
166 Jul 16 13:14:54.802 INFO request completed, latency_us: 600, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: 091492bf-51a1-4201-88d7-e3bfaf4370fc, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
167 Jul 16 13:14:55.803 INFO request completed, latency_us: 726, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: a289a4ba-6dc1-4397-9922-c614f46e9121, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
168 Jul 16 13:14:56.802 INFO request completed, latency_us: 594, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: e5ff569d-2499-4e87-96f9-450ad36fc2c5, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
169 Jul 16 13:14:57.803 INFO request completed, latency_us: 595, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: 3937e69e-6044-4195-84ce-f38236659125, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
170 Jul 16 13:14:58.802 INFO request completed, latency_us: 589, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: a48ba8c5-daa9-4c8d-b4c3-98a4739697d5, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
171 Jul 16 13:14:59.802 INFO request completed, latency_us: 731, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: cad7bba8-3b4f-4467-93ea-28604b53ba7e, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
172 Jul 16 13:15:00.803 INFO request completed, latency_us: 897, response_code: 200, uri: /18823c2d-71d6-4785-a63c-2b91da01f424, method: GET, req_id: ab875972-d7c2-4028-a26b-220602a3b842, remote_addr: [fd00:1122:3344:101::d]:60965, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874