Max level shown:
1 [ Jul 29 01:11:03 Disabled. ]
2 [ Jul 29 01:11:03 Rereading configuration. ]
3 [ Jul 29 01:11:04 Enabled. ]
4 [ Jul 29 01:11:04 Rereading configuration. ]
5 [ Jul 29 01:11:04 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/mg-ddm/pkg/ddm_method_script.sh &"). ]
6 [ Jul 29 01:11:04 Method "start" exited with status 0. ]
72026-07-29T01:11:04.171ZINFOslog-rs: signal handler waiting for context
82026-07-29T01:11:04.192ZERROslog-rs: failed to start stats server: underlay address not found
92026-07-29T01:11:04.192ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
102026-07-29T01:11:04.192ZINFOslog-rs: signal handler waiting got context
112026-07-29T01:11:04.193ZINFOslog-rs: admin: listening on [::1]:8000
122026-07-29T01:11:05.192ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
132026-07-29T01:11:06.192ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
142026-07-29T01:11:07.192ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
152026-07-29T01:11:08.192ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
162026-07-29T01:11:09.214ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
172026-07-29T01:11:10.193ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
182026-07-29T01:11:11.193ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
192026-07-29T01:11:12.193ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
202026-07-29T01:11:13.193ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
212026-07-29T01:11:14.193ZWARNslog-rs: [] failed to get IPv6 address for interface tfportrear0_0/ll: not found: tfportrear0_0/ll
222026-07-29T01:11:15.194ZINFOslog-rs: [tfportrear0_0] sm initialized with addr fe80::f800:a7ff:fe3a:4bba on if tfportrear0_0 index 3
232026-07-29T01:11:15.194ZDEBGslog-rs: [tfportrear0_0] starting discovery handler
242026-07-29T01:11:15.194ZINFOslog-rs: [tfportrear0_0] nbr is fe80::2ca4:32ff:feaf:dfaf@buskin server
252026-07-29T01:11:15.194ZDEBGslog-rs: [tfportrear0_0] transition solicit -> exchange
262026-07-29T01:11:15.195ZINFOslog-rs: [tfportrear0_0] exchange: listening on [fe80::f800:a7ff:fe3a:4bba]:56797
272026-07-29T01:11:15.195ZINFOslog-rs: [tfportrear0_0] waiting for exchange server to start
28 [ Jul 29 01:11:15 Rereading configuration. ]
29 [ Jul 29 01:11:15 Executing refresh method ("/opt/oxide/mg-ddm/pkg/refresh_ctrun_child.sh -c 213"). ]
30 + CONTRACT=''
31 + getopts c: o
32 + CONTRACT=213
33 + getopts c: o
34 + [[ -n 213 ]]
35 + pgrep -c 213
36 + PARENT_PID=2296
37 + [[ -n 2296 ]]
38 + pgrep -P 2296
39 + CHILD_PID=2299
40 + [[ -n 2299 ]]
41 + kill -USR1 2299
422026-07-29T01:11:15.648ZINFOslog-rs: handling smf refresh
43 [ Jul 29 01:11:15 Method "refresh" exited with status 0. ]
442026-07-29T01:11:15.649ZINFOslog-rs: starting stats server on smf refresh
45 Jul 29 01:11:15.650 DEBG registered DTrace probes
46 Jul 29 01:11:15.651 DEBG registered endpoint, versions: all, path: /{producer_id}, method: GET, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
47 Jul 29 01:11:15.651 INFO listening, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:335
48 Jul 29 01:11:15.651 DEBG successfully registered DTrace USDT probes, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot
49 Jul 29 01:11:15.651 DEBG Nexus IP not provided, will use DNS to resolve it
50 Jul 29 01:11:15.651 INFO new DNS resolver, addresses: [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53], component: internal-dns-resolver, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/internal-dns/resolver/src/resolver.rs:144
51 Jul 29 01:11:15.651 DEBG starting producer registration task
52 Jul 29 01:11:15.652 INFO starting oximeter metric producer server, interval: 1s, address: [fd00:1122:3344:101::2]:8001, producer_id: 3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:280
53 Jul 29 01:11:15.652 DEBG registering as oximeter producer with Nexus, registration_count: 0, component: producer-registration-task
54 Jul 29 01:11:15.653 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 1.816481ms, count: 0, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
55 Jul 29 01:11:15.838 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 186.451659ms, count: 1, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
56 Jul 29 01:11:16.246 WARN failed to register with Nexus, will retry, error: proto error: no records found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, delay: 594.453037ms, count: 2, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
572026-07-29T01:11:16.844ZINFOslog-rs: advertise tunnel: { TunnelOrigin { overlay_prefix: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), boundary_addr: fdd9:3e55:65e0:653d::1, vni: 99, metric: 18446744073709551615, }, }
582026-07-29T01:11:17.197ZWARNslog-rs: [tfportrear0_0] exchange pull: hyper client error: client error (Connect)
592026-07-29T01:11:17.198ZINFOslog-rs: [tfportrear0_0] sending 5 routes to dendrite
602026-07-29T01:11:17.198ZDEBGslog-rs: [tfportrear0_0] sending to dpd host=localhost port=12224
612026-07-29T01:11:17.200ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:2:: -> fe80::2ca4:32ff:feaf:dfaf on port Rear(Rear("rear0"))/0
622026-07-29T01:11:17.202ZINFOslog-rs: [tfportrear0_0] adding route fdb0:18c0:4d0d:9fb2:: -> fe80::2ca4:32ff:feaf:dfaf on port Rear(Rear("rear0"))/0
632026-07-29T01:11:17.202ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:3:: -> fe80::2ca4:32ff:feaf:dfaf on port Rear(Rear("rear0"))/0
642026-07-29T01:11:17.203ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:101:: -> fe80::2ca4:32ff:feaf:dfaf on port Rear(Rear("rear0"))/0
652026-07-29T01:11:17.203ZINFOslog-rs: [tfportrear0_0] adding route fd00:1122:3344:1:: -> fe80::2ca4:32ff:feaf:dfaf on port Rear(Rear("rear0"))/0
662026-07-29T01:11:17.203ZINFOslog-rs: [tfportrear0_0] removing routes 0 from dendrite
672026-07-29T01:11:17.204ZDEBGslog-rs: [tfportrear0_0] redistributing update to 0 peers
68 Jul 29 01:11:17.417 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1f14d2b3-fdd7-4bc5-97a2-1d7d25fde44f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91282035-0273-4188-a9b4-a4f9c97f03d8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1284, tv_nsec: 117256180 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
69 Jul 29 01:11:17.418 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
70 Jul 29 01:11:17.419 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
71 Jul 29 01:11:32.420 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
72 Jul 29 01:11:32.420 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 16.768509033s, count: 3, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
73 Jul 29 01:11:34.571 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1f14d2b3-fdd7-4bc5-97a2-1d7d25fde44f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91282035-0273-4188-a9b4-a4f9c97f03d8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1301, tv_nsec: 270963482 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
74 Jul 29 01:11:34.571 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
75 Jul 29 01:11:34.572 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
762026-07-29T01:11:41.257ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2ca4:32ff:feaf:dfaf missed solicit interval
77 Jul 29 01:11:49.618 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
78 Jul 29 01:11:49.619 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 33.922209115s, count: 4, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
79 Jul 29 01:11:53.271 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1f14d2b3-fdd7-4bc5-97a2-1d7d25fde44f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91282035-0273-4188-a9b4-a4f9c97f03d8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1319, tv_nsec: 916724588 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
80 Jul 29 01:11:53.272 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
81 Jul 29 01:11:53.272 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
822026-07-29T01:12:03.319ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2ca4:32ff:feaf:dfaf missed solicit interval
83 Jul 29 01:12:08.314 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
84 Jul 29 01:12:08.314 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 52.566930843s, count: 5, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
85 Jul 29 01:12:17.061 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1f14d2b3-fdd7-4bc5-97a2-1d7d25fde44f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91282035-0273-4188-a9b4-a4f9c97f03d8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1343, tv_nsec: 642412653 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
86 Jul 29 01:12:17.062 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
87 Jul 29 01:12:17.063 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
88 Jul 29 01:12:32.105 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
89 Jul 29 01:12:32.105 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 76.293484083s, count: 6, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
90 Jul 29 01:12:43.247 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1f14d2b3-fdd7-4bc5-97a2-1d7d25fde44f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91282035-0273-4188-a9b4-a4f9c97f03d8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1369, tv_nsec: 757374401 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
91 Jul 29 01:12:43.248 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
92 Jul 29 01:12:43.249 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
932026-07-29T01:12:52.007ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2ca4:32ff:feaf:dfaf missed solicit interval
942026-07-29T01:12:56.193ZWARNslog-rs: [tfportrear0_0] neighbor buskin@fe80::2ca4:32ff:feaf:dfaf missed solicit interval
95 Jul 29 01:12:58.302 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: TimedOut }), component: producer-registration-task
96 Jul 29 01:12:58.308 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out, delay: 102.424128088s, count: 7, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
97 Jul 29 01:13:39.146 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1f14d2b3-fdd7-4bc5-97a2-1d7d25fde44f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91282035-0273-4188-a9b4-a4f9c97f03d8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1425, tv_nsec: 603700489 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
98 Jul 29 01:13:39.147 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
99 Jul 29 01:13:39.158 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
100 Jul 29 01:13:39.165 DEBG client response, result: Err(reqwest::Error { kind: Request, url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", [fd00:1122:3344:101::a]:12221, Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) }), component: producer-registration-task
101 Jul 29 01:13:39.165 WARN failed to register with Nexus, will retry, error: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): client error (Connect): tcp connect error: Connection refused (os error 146), delay: 143.272516397s, count: 8, component: producer-registration-task, file: /home/build/.cargo/git/checkouts/omicron-e0115f488d1c1d58/d6fba55/oximeter/producer/src/lib.rs:376
102 Jul 29 01:14:15.427 DEBG lookup_socket_v6 srv, response: SrvLookup(Lookup { query: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("1f14d2b3-fdd7-4bc5-97a2-1d7d25fde44f.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("91282035-0273-4188-a9b4-a4f9c97f03d8.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("_nexus._tcp.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: SRV(SRV { priority: 0, weight: 0, port: 12221, target: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal.") }) }, Record { name_labels: Name("eec9f85c-4b85-4275-af8c-f39029b7f38e.host.control-plane.oxide.internal."), dns_class: IN, ttl: 0, rdata: AAAA(AAAA(fd00:1122:3344:101::c)) }], valid_until: Instant { tv_sec: 1461, tv_nsec: 886127273 } }), dns_name: _nexus._tcp.control-plane.oxide.internal, component: internal-dns-resolver
103 Jul 29 01:14:15.428 DEBG will register with Nexus at [fd00:1122:3344:101::a]:12221, component: producer-registration-task
104 Jul 29 01:14:15.429 DEBG client request, body: Some(Body), uri: http://[fd00:1122:3344:101::a]:12221/metrics/producers, method: POST, component: producer-registration-task
105 Jul 29 01:14:15.444 DEBG client response, result: Ok(Response { url: "http://[fd00:1122:3344:101::a]:12221/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "aa94b89c-020d-4966-a07d-5a0536128dc2", "content-length": "41", "date": "Wed, 29 Jul 2026 01:14:15 GMT"} }), component: producer-registration-task
106 Jul 29 01:14:15.445 DEBG successfully registered with Nexus, pausing until time to renew lease, wait_period: 150s, lease_duration: 600s, component: producer-registration-task
107 Jul 29 01:14:26.196 INFO accepted connection, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:1057
108 Jul 29 01:14:26.203 INFO request completed, latency_us: 6771, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: f364f597-a8bd-4520-9d0e-d25617b3a74f, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
109 Jul 29 01:14:27.197 INFO request completed, latency_us: 1727, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 257cfa91-527b-41fe-ad92-53666e009572, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
110 Jul 29 01:14:28.196 INFO request completed, latency_us: 714, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: cfca231c-9d1e-46fb-9705-9cce406cf0ea, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
111 Jul 29 01:14:29.196 INFO request completed, latency_us: 721, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 5e178dfb-4973-45f1-9ee5-f555b5fc1427, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
112 Jul 29 01:14:30.195 INFO request completed, latency_us: 595, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: d9a0e194-8751-4e73-8be5-8246b92d2ac5, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
113 Jul 29 01:14:31.196 INFO request completed, latency_us: 572, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: bfa4f272-adfd-47c1-a316-930f551aa10d, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
114 Jul 29 01:14:32.196 INFO request completed, latency_us: 583, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 46fb8503-e33f-470e-b787-e81d54e0010d, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
115 Jul 29 01:14:33.197 INFO request completed, latency_us: 834, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: b5b0488c-a275-40ad-8bce-341c383b0815, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
116 Jul 29 01:14:34.196 INFO request completed, latency_us: 732, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 0ee70596-a55c-437f-b853-9ac2ababac16, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
117 Jul 29 01:14:35.196 INFO request completed, latency_us: 599, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: e6f53903-1c14-4d88-80a0-5331a455cccb, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
118 Jul 29 01:14:36.196 INFO request completed, latency_us: 734, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: bc828d90-f6a7-4939-a2d9-87928a0a2d65, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
119 Jul 29 01:14:37.196 INFO request completed, latency_us: 827, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 1cabb935-2547-4912-ac46-0f6f55e66599, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
120 Jul 29 01:14:38.196 INFO request completed, latency_us: 853, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 7d4aa3a1-46a4-403f-84aa-0747b4ff0136, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
121 Jul 29 01:14:39.196 INFO request completed, latency_us: 735, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: ffa50d3a-9f46-4cdd-b9e1-e0ee2225e271, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
122 Jul 29 01:14:40.196 INFO request completed, latency_us: 726, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 10a611bc-40c0-4c20-9719-27c7c41898e0, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
123 Jul 29 01:14:41.196 INFO request completed, latency_us: 598, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 4963b31e-ed58-4984-994d-1a744f2e6484, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
124 Jul 29 01:14:42.196 INFO request completed, latency_us: 884, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 66b256b2-0e11-40a1-aa22-cbffcc4f0ca4, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
125 Jul 29 01:14:43.196 INFO request completed, latency_us: 620, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 1447b017-bdef-466a-bf2e-5a611680dede, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
126 Jul 29 01:14:44.196 INFO request completed, latency_us: 1089, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 1ae1384f-eb5f-473a-bed9-8ebc41904a0d, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
127 Jul 29 01:14:45.197 INFO request completed, latency_us: 660, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: bab497db-7b9c-4f87-9853-cb337387944e, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
128 Jul 29 01:14:46.196 INFO request completed, latency_us: 651, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 3a73e928-932b-4f59-8c4b-70171d6bacae, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
129 Jul 29 01:14:47.196 INFO request completed, latency_us: 639, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 5371cff7-8ec1-4583-a6af-e55334be9e4e, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
130 Jul 29 01:14:48.196 INFO request completed, latency_us: 738, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 53602de4-7439-4501-9e97-11a3318ff62c, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
131 Jul 29 01:14:49.196 INFO request completed, latency_us: 584, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 2742a6cb-e631-4159-a9c2-1aefd1345825, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
132 Jul 29 01:14:50.196 INFO request completed, latency_us: 624, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 1f9efdf0-67dc-494a-ad07-ed89ca1b438a, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
133 Jul 29 01:14:51.196 INFO request completed, latency_us: 627, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 65bed497-7bff-478f-9190-709eac242b1f, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
134 Jul 29 01:14:52.196 INFO request completed, latency_us: 605, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: b6002428-3e68-4b2b-a069-24bcd8eccf73, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
135 Jul 29 01:14:53.197 INFO request completed, latency_us: 725, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: afa28cb2-0181-4f28-aedb-a68286e37ba4, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
136 Jul 29 01:14:54.196 INFO request completed, latency_us: 608, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 7d043f5d-c7bb-4076-88df-4ea0481c350d, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
137 Jul 29 01:14:55.196 INFO request completed, latency_us: 603, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 1ccc7feb-bac5-4cc9-9d31-ad8139d5c2ad, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
138 Jul 29 01:14:56.196 INFO request completed, latency_us: 588, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 51aea81a-835a-49b3-8799-5194fc77f5a1, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
139 Jul 29 01:14:57.197 INFO request completed, latency_us: 720, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 570fe4d7-abe6-4bc6-a667-29f783c28caa, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
140 Jul 29 01:14:58.197 INFO request completed, latency_us: 842, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 41d053c2-f297-4d16-bf95-bdabd6332b3a, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
141 Jul 29 01:14:59.197 INFO request completed, latency_us: 724, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: b4c4b0f0-4109-4ccb-86ea-c7c9bcf0fc3f, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874
142 Jul 29 01:15:00.196 INFO request completed, latency_us: 583, response_code: 200, uri: /3a949c8d-0c9c-4e51-a0db-fbe1bf055f38, method: GET, req_id: 3df9a9f4-7c4e-4b96-be2d-935ef2bd6b7a, remote_addr: [fd00:1122:3344:101::d]:44527, local_addr: [fd00:1122:3344:101::2]:8001, component: dropshot, file: /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.17.1/src/server.rs:874