Max level shown:
1 [ Jul 2 03:42:52 Enabled. ]
2 [ Jul 2 03:42:52 Rereading configuration. ]
3 [ Jul 2 03:42:53 Rereading configuration. ]
4 [ Jul 2 03:42:54 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Jul 2 03:42:54 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-02T03:42:54.552ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-07-02T03:42:54.553ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-02T03:42:54.573ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-02T03:42:54.573ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-02T03:42:54.573ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-07-02T03:42:54.574ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-07-02T03:42:54.671ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8320, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:28292 req_id = 8f394f97-ab40-43c0-8bca-37f0d12f919c
152026-07-02T03:42:54.846ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:52688 req_id = 93eb54dd-9211-4873-ab16-4c3b40c45737
162026-07-02T03:42:55.061ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:63795
172026-07-02T03:42:55.062ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 443692ee-1e86-4147-8a1d-f515d15f4713
182026-07-02T03:42:55.062ZINFOdns-server (store): pruning trees for generations newer than 0
192026-07-02T03:42:55.074ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
202026-07-02T03:42:55.074ZINFOdns-server (store): updated generation new_generation = 1 req_id = 443692ee-1e86-4147-8a1d-f515d15f4713
212026-07-02T03:42:55.074ZINFOdns-server (http): request completed latency_us = 12772 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:63795 req_id = 443692ee-1e86-4147-8a1d-f515d15f4713 response_code = 204 uri = /config
222026-07-02T03:43:07.924ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13904, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:36367 req_id = 0750424c-a119-4bc4-a01e-68fa4ed54001
232026-07-02T03:43:07.928ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45084, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:48596 req_id = 83ee421e-b40b-412e-8636-67637f52a162
242026-07-02T03:43:07.933ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39414, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:50774 req_id = 4a0f7416-06aa-4ed7-b475-04dd4e8fd104
252026-07-02T03:44:19.327ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28659, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52369 req_id = 5ed29eda-f218-4c3e-9375-f13bed56128c
262026-07-02T03:44:20.963ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:40000 req_id = cd70db47-0bf4-4290-96ae-5bb8d0687ab7
272026-07-02T03:44:20.964ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:55073 req_id = 55be7099-b2a1-4399-ad87-883c45365d98
282026-07-02T03:44:28.916ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64035, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:43586 req_id = 4d376261-6850-4f71-b7c8-4002228195ff
292026-07-02T03:44:28.952ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41445, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:59563 req_id = ed7f02a4-247b-4577-903f-259205df06e6
302026-07-02T03:44:29.317ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59438, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55963 req_id = b9ae04b1-8b3c-4ac5-9d86-7dd6af57b5f2
312026-07-02T03:44:29.559ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15751, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:8460 req_id = 750ae83a-18f1-480e-8d3b-04f344fa3258
322026-07-02T03:44:39.319ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39097, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51992 req_id = 7ff698fc-d449-409d-aef3-878ffdc75720
332026-07-02T03:44:49.321ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39149, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52574 req_id = 356de610-f233-4770-a884-9ddc90886705
342026-07-02T03:44:59.323ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24601, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49778 req_id = 4ce3afb7-ff7c-4a46-8538-932c86fc8375
352026-07-02T03:45:00.293ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23838, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_dcb336db-e940-41be-8947-64e008af2100."), ), original: Query { name: Name("oxz_ntp_dcb336db-e940-41be-8947-64e008af2100."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 100, 99, 98, 51, 51, 54, 100, 98, 45, 101, 57, 52, 48, 45, 52, 49, 98, 101, 45, 56, 57, 52, 55, 45, 54, 52, 101, 48, 48, 56, 97, 102, 50, 49, 48, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_dcb336db-e940-41be-8947-64e008af2100." peer_addr = [fd00:1122:3344:101::1b]:59403 req_id = d723247d-ceb0-4950-8376-9ca3d06030ca
362026-07-02T03:45:00.309ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:58835 req_id = a6f08683-ca33-4bd4-9c7b-a083149d57d6
372026-07-02T03:45:00.310ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:38380 req_id = dc79905a-5f6d-4d00-98be-65caeb4305be
382026-07-02T03:45:00.463ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9321, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:45839 req_id = 24b62dd5-63f1-4899-a6bf-9f0229422ce8
392026-07-02T03:45:00.464ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9321, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:50266 req_id = e58386c3-07ad-4483-8241-537873b33830
402026-07-02T03:45:00.577ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45455, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), ), original: Query { name: Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 55, 49, 48, 51, 98, 51, 57, 45, 48, 99, 102, 53, 45, 52, 99, 56, 54, 45, 56, 100, 57, 100, 45, 100, 48, 50, 99, 52, 51, 55, 56, 101, 53, 55, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576." peer_addr = [fd00:1122:3344:101::4]:43692 req_id = 0d6e9032-a87c-444b-ac94-22ad602f7d17
412026-07-02T03:45:00.578ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45455, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), ), original: Query { name: Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 55, 49, 48, 51, 98, 51, 57, 45, 48, 99, 102, 53, 45, 52, 99, 56, 54, 45, 56, 100, 57, 100, 45, 100, 48, 50, 99, 52, 51, 55, 56, 101, 53, 55, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576." peer_addr = [fd00:1122:3344:101::4]:60783 req_id = 38ffddc9-b176-42d9-94f6-6e6d7f035a8d
422026-07-02T03:45:00.653ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), ), original: Query { name: Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 55, 55, 57, 48, 57, 48, 99, 45, 102, 48, 97, 57, 45, 52, 98, 55, 52, 45, 57, 99, 48, 48, 45, 51, 53, 100, 101, 53, 51, 51, 101, 98, 49, 52, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d." peer_addr = [fd00:1122:3344:101::5]:60847 req_id = 3ba5e6aa-f1f4-4327-930e-194e4719900a
432026-07-02T03:45:00.654ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), ), original: Query { name: Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 55, 55, 57, 48, 57, 48, 99, 45, 102, 48, 97, 57, 45, 52, 98, 55, 52, 45, 57, 99, 48, 48, 45, 51, 53, 100, 101, 53, 51, 51, 101, 98, 49, 52, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d." peer_addr = [fd00:1122:3344:101::5]:37541 req_id = 30059dfc-4b29-4246-81bd-31f5448794f8
442026-07-02T03:45:00.681ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49561, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), ), original: Query { name: Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 102, 52, 50, 101, 50, 50, 50, 45, 52, 100, 100, 54, 45, 52, 48, 99, 52, 45, 56, 48, 57, 52, 45, 52, 48, 54, 98, 102, 100, 57, 55, 101, 51, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394." peer_addr = [fd00:1122:3344:101::3]:37289 req_id = 9c070414-fdb7-469f-8f06-ba2906526053
452026-07-02T03:45:00.682ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49561, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), ), original: Query { name: Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 102, 52, 50, 101, 50, 50, 50, 45, 52, 100, 100, 54, 45, 52, 48, 99, 52, 45, 56, 48, 57, 52, 45, 52, 48, 54, 98, 102, 100, 57, 55, 101, 51, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394." peer_addr = [fd00:1122:3344:101::3]:51828 req_id = 83f7f584-6782-4dba-8e48-a37ce08fdc9c
462026-07-02T03:45:00.684ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10081, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), ), original: Query { name: Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 50, 54, 100, 54, 49, 101, 53, 45, 102, 101, 51, 98, 45, 52, 51, 51, 100, 45, 57, 53, 50, 48, 45, 54, 48, 48, 101, 99, 102, 53, 48, 57, 49, 48, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f." peer_addr = [fd00:1122:3344:101::6]:46932 req_id = a8e556ec-ef2d-4e0d-80fc-2172df7073d0
472026-07-02T03:45:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10081, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), ), original: Query { name: Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 50, 54, 100, 54, 49, 101, 53, 45, 102, 101, 51, 98, 45, 52, 51, 51, 100, 45, 57, 53, 50, 48, 45, 54, 48, 48, 101, 99, 102, 53, 48, 57, 49, 48, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f." peer_addr = [fd00:1122:3344:101::6]:62144 req_id = 782a64b6-1055-4754-aebe-58d4d7633d21
482026-07-02T03:45:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7086, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), ), original: Query { name: Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 100, 100, 101, 53, 50, 102, 49, 45, 102, 99, 49, 49, 45, 52, 49, 101, 55, 45, 98, 49, 56, 98, 45, 102, 49, 53, 49, 48, 57, 98, 56, 99, 49, 56, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186." peer_addr = [fd00:1122:3344:101::7]:35600 req_id = 59f67b6c-eb38-4aee-a2d1-9e918eee36ec
492026-07-02T03:45:00.686ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7086, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), ), original: Query { name: Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 100, 100, 101, 53, 50, 102, 49, 45, 102, 99, 49, 49, 45, 52, 49, 101, 55, 45, 98, 49, 56, 98, 45, 102, 49, 53, 49, 48, 57, 98, 56, 99, 49, 56, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186." peer_addr = [fd00:1122:3344:101::7]:62079 req_id = 13199ce0-f2ad-4884-8200-3fe4adf38eba