Max level shown:
1 [ Jul 23 06:25:20 Enabled. ]
2 [ Jul 23 06:25:20 Rereading configuration. ]
3 [ Jul 23 06:25:21 Rereading configuration. ]
4 [ Jul 23 06:25:22 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:1::1]:5353 --dns-address [fd00:1122:3344:1::1]:53 &"). ]
5 [ Jul 23 06:25:22 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-23T06:25:22.253ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:1::1]:53 }
92026-07-23T06:25:22.253ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-23T06:25:22.275ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-23T06:25:22.275ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-23T06:25:22.275ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:1::1]:53
132026-07-23T06:25:22.275ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:1::1]:5353
142026-07-23T06:25:22.282ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27795, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:52951 req_id = 52f1d737-d701-44d7-8d4c-02624481486b
152026-07-23T06:25:22.401ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28606, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:58523 req_id = 10c4eb3f-b128-49c6-9178-5f2883af0143
162026-07-23T06:25:22.565ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22856, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:16233 req_id = 8b028e9c-071d-4aa1-af01-63c2da67c245
172026-07-23T06:25:22.618ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:1::2]:44181
182026-07-23T06:25:22.619ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = eb956cc9-757b-47b2-a751-d376d4507507
192026-07-23T06:25:22.619ZINFOdns-server (store): pruning trees for generations newer than 0
202026-07-23T06:25:22.631ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
212026-07-23T06:25:22.631ZINFOdns-server (store): updated generation new_generation = 1 req_id = eb956cc9-757b-47b2-a751-d376d4507507
222026-07-23T06:25:22.631ZINFOdns-server (http): request completed latency_us = 13241 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:1::2]:44181 req_id = eb956cc9-757b-47b2-a751-d376d4507507 response_code = 204 uri = /config
232026-07-23T06:25:37.503ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34656, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:56828 req_id = a827a7a9-e6a4-4ee5-b033-043a2046b98e
242026-07-23T06:25:37.508ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6415, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:55558 req_id = 662c1dc8-86c6-4ad0-8bcc-31ce09631026
252026-07-23T06:25:37.513ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27623, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:64866 req_id = 21700ada-ae37-473b-8cc5-1dae625b659d
262026-07-23T06:27:38.650ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35410, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:39663 req_id = 3078b0f8-290b-4b73-8dc9-12034802118d
272026-07-23T06:27:38.655ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:64789 req_id = f00b6e6a-c895-4b36-b31f-6f26cd83bca9
282026-07-23T06:27:38.659ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62913, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:55100 req_id = f5c4f58d-6a19-41a4-bc8f-95bf0053a72a
292026-07-23T06:29:10.202ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49283, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60593 req_id = dace44f0-3bc1-48dd-8831-b784233e74fa
302026-07-23T06:29:13.836ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25968, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), ), original: Query { name: Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 53, 99, 57, 56, 54, 48, 51, 100, 45, 54, 100, 102, 50, 45, 52, 102, 102, 54, 45, 57, 102, 52, 98, 45, 52, 101, 98, 101, 98, 48, 52, 53, 52, 51, 99, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3." peer_addr = [fd00:1122:3344:101::e]:57914 req_id = bda6c811-22ae-43ad-a013-553569b58022
312026-07-23T06:29:13.852ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25968, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), ), original: Query { name: Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 53, 99, 57, 56, 54, 48, 51, 100, 45, 54, 100, 102, 50, 45, 52, 102, 102, 54, 45, 57, 102, 52, 98, 45, 52, 101, 98, 101, 98, 48, 52, 53, 52, 51, 99, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3." peer_addr = [fd00:1122:3344:101::e]:40440 req_id = b01f8735-bdfc-46c4-838c-8f0d2b2651eb
322026-07-23T06:29:19.166ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19874, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:20647 req_id = fcad8f7c-7a60-42a7-bf49-f46d32c36734
332026-07-23T06:29:19.170ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34984, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:29573 req_id = c14ba561-2905-49e1-81e9-a345feeb6da9
342026-07-23T06:29:20.185ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1858, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58778 req_id = 797a93ad-e7e8-46f0-b57f-63374990faf4
352026-07-23T06:29:24.837ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6927, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:35834 req_id = 0d4c929c-b637-4124-bd2d-7fcca23f4778
362026-07-23T06:29:25.435ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:53147
372026-07-23T06:29:25.512ZINFOdns-server (http): request completed latency_us = 39180 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:53147 req_id = 1f32210a-f3f8-404d-ae88-949452045284 response_code = 200 uri = /config
382026-07-23T06:29:26.367ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32740, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:61101 req_id = 520a3e92-68d2-4636-a151-eb92a2043b74
392026-07-23T06:29:26.556ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:45701
402026-07-23T06:29:26.566ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 71b3ca00-3041-4cfe-8cef-0f7e24470c62
412026-07-23T06:29:26.566ZINFOdns-server (store): updated generation new_generation = 1 req_id = 71b3ca00-3041-4cfe-8cef-0f7e24470c62
422026-07-23T06:29:26.566ZINFOdns-server (http): request completed latency_us = 9722 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:45701 req_id = 71b3ca00-3041-4cfe-8cef-0f7e24470c62 response_code = 204 uri = /config
432026-07-23T06:29:26.647ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:46225
442026-07-23T06:29:26.647ZINFOdns-server (http): request completed latency_us = 243 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:46225 req_id = f7ad5a58-92aa-4570-b793-f552328a9322 response_code = 200 uri = /config
452026-07-23T06:29:27.560ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 575, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:37806 req_id = 4ab55547-81e4-4cec-a35c-9bfb6d0fb26e
462026-07-23T06:29:27.709ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:59579
472026-07-23T06:29:27.710ZINFOdns-server (http): request completed latency_us = 225 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:59579 req_id = 80f40ea6-1c3e-4a5e-a9d8-f04ec2ed07a9 response_code = 200 uri = /config
482026-07-23T06:29:29.334ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16492, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:15034 req_id = 46e7880b-626d-4e12-823a-1c017fe85eac
492026-07-23T06:29:29.433ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:48607
502026-07-23T06:29:29.433ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = a1e521a2-09f7-4d0d-924b-f9fb9e3c16f1
512026-07-23T06:29:29.433ZINFOdns-server (store): updated generation new_generation = 1 req_id = a1e521a2-09f7-4d0d-924b-f9fb9e3c16f1
522026-07-23T06:29:29.433ZINFOdns-server (http): request completed latency_us = 324 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:48607 req_id = a1e521a2-09f7-4d0d-924b-f9fb9e3c16f1 response_code = 204 uri = /config
532026-07-23T06:29:29.673ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:56281
542026-07-23T06:29:29.674ZINFOdns-server (http): request completed latency_us = 489 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:56281 req_id = 80f156ba-a178-4cb5-941c-01187ddabffc response_code = 200 uri = /config
552026-07-23T06:29:30.186ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5849, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55967 req_id = a1e3d108-353f-408d-8091-394823c09f9e
562026-07-23T06:29:30.770ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51106, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:58461 req_id = 9b86e986-cb0b-4fe5-b291-29fb0dfa6e28
572026-07-23T06:29:30.888ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:59442
582026-07-23T06:29:30.888ZINFOdns-server (http): request completed latency_us = 192 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:59442 req_id = f72ef889-829c-43da-83ae-63284a9bede6 response_code = 200 uri = /config
592026-07-23T06:29:31.253ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43584, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:22010 req_id = b6c7b3d4-6b49-4f6a-b257-a005eb133bf0
602026-07-23T06:29:31.363ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:41317
612026-07-23T06:29:31.363ZINFOdns-server (http): request completed latency_us = 243 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:41317 req_id = c619eb4a-b232-43fa-9cc6-fc02aaa4ac29 response_code = 200 uri = /config
622026-07-23T06:29:38.775ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37895, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:33113 req_id = 00fb1eb3-1448-4c84-8aa0-0217b43b2993
632026-07-23T06:29:38.780ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24835, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:47490 req_id = 6458959c-ae66-49fe-acd4-0e589086a37a
642026-07-23T06:29:38.784ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3442, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:37850 req_id = 40589846-604c-415b-841e-37ea4a474629
652026-07-23T06:29:40.188ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51953, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55257 req_id = dc0385fe-7808-4185-8a08-ae791f30b62c
662026-07-23T06:29:50.191ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56169, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64075 req_id = 99aaaad6-bd7e-4f5f-a5e6-817c2c974417
672026-07-23T06:30:00.086ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_4f1ebf7d-7861-4655-b522-fe3100321e6d."), ), original: Query { name: Name("oxz_cockroachdb_4f1ebf7d-7861-4655-b522-fe3100321e6d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 52, 102, 49, 101, 98, 102, 55, 100, 45, 55, 56, 54, 49, 45, 52, 54, 53, 53, 45, 98, 53, 50, 50, 45, 102, 101, 51, 49, 48, 48, 51, 50, 49, 101, 54, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_4f1ebf7d-7861-4655-b522-fe3100321e6d." peer_addr = [fd00:1122:3344:101::5]:33647 req_id = d9b77f32-c0fa-4124-b866-36b96e8f7edb
682026-07-23T06:30:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_4f1ebf7d-7861-4655-b522-fe3100321e6d."), ), original: Query { name: Name("oxz_cockroachdb_4f1ebf7d-7861-4655-b522-fe3100321e6d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 52, 102, 49, 101, 98, 102, 55, 100, 45, 55, 56, 54, 49, 45, 52, 54, 53, 53, 45, 98, 53, 50, 50, 45, 102, 101, 51, 49, 48, 48, 51, 50, 49, 101, 54, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_4f1ebf7d-7861-4655-b522-fe3100321e6d." peer_addr = [fd00:1122:3344:101::5]:37007 req_id = 6ea66217-fde6-4ecd-94b3-19f19ccac778
692026-07-23T06:30:00.136ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2550, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), ), original: Query { name: Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 53, 99, 57, 56, 54, 48, 51, 100, 45, 54, 100, 102, 50, 45, 52, 102, 102, 54, 45, 57, 102, 52, 98, 45, 52, 101, 98, 101, 98, 48, 52, 53, 52, 51, 99, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3." peer_addr = [fd00:1122:3344:101::e]:57453 req_id = 2d8f98f6-5d46-4f55-a7fb-6fdcbaf45df6
702026-07-23T06:30:00.137ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2550, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), ), original: Query { name: Name("oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 53, 99, 57, 56, 54, 48, 51, 100, 45, 54, 100, 102, 50, 45, 52, 102, 102, 54, 45, 57, 102, 52, 98, 45, 52, 101, 98, 101, 98, 48, 52, 53, 52, 51, 99, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_5c98603d-6df2-4ff6-9f4b-4ebeb04543c3." peer_addr = [fd00:1122:3344:101::e]:64956 req_id = 66d08bc6-faae-4203-a63d-90b31b6df288
712026-07-23T06:30:00.170ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65282, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_038c380e-1c60-4465-8e97-f1b5e7470d0a."), ), original: Query { name: Name("oxz_cockroachdb_038c380e-1c60-4465-8e97-f1b5e7470d0a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 51, 56, 99, 51, 56, 48, 101, 45, 49, 99, 54, 48, 45, 52, 52, 54, 53, 45, 56, 101, 57, 55, 45, 102, 49, 98, 53, 101, 55, 52, 55, 48, 100, 48, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_038c380e-1c60-4465-8e97-f1b5e7470d0a." peer_addr = [fd00:1122:3344:101::4]:60588 req_id = 771a66e8-ee1a-4f16-b1f8-7a4631e551c6
722026-07-23T06:30:00.171ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65282, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_038c380e-1c60-4465-8e97-f1b5e7470d0a."), ), original: Query { name: Name("oxz_cockroachdb_038c380e-1c60-4465-8e97-f1b5e7470d0a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 51, 56, 99, 51, 56, 48, 101, 45, 49, 99, 54, 48, 45, 52, 52, 54, 53, 45, 56, 101, 57, 55, 45, 102, 49, 98, 53, 101, 55, 52, 55, 48, 100, 48, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_038c380e-1c60-4465-8e97-f1b5e7470d0a." peer_addr = [fd00:1122:3344:101::4]:48724 req_id = 68356f39-bf5b-4910-a690-8fd7818b7ed9
732026-07-23T06:30:00.181ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14602, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_1f4d04d5-0620-435e-8421-435babc6ef77."), ), original: Query { name: Name("oxz_cockroachdb_1f4d04d5-0620-435e-8421-435babc6ef77."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 102, 52, 100, 48, 52, 100, 53, 45, 48, 54, 50, 48, 45, 52, 51, 53, 101, 45, 56, 52, 50, 49, 45, 52, 51, 53, 98, 97, 98, 99, 54, 101, 102, 55, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_1f4d04d5-0620-435e-8421-435babc6ef77." peer_addr = [fd00:1122:3344:101::3]:37054 req_id = 0b9d4716-fae6-4150-bf20-7bac3b2760a3
742026-07-23T06:30:00.182ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14602, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_1f4d04d5-0620-435e-8421-435babc6ef77."), ), original: Query { name: Name("oxz_cockroachdb_1f4d04d5-0620-435e-8421-435babc6ef77."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 102, 52, 100, 48, 52, 100, 53, 45, 48, 54, 50, 48, 45, 52, 51, 53, 101, 45, 56, 52, 50, 49, 45, 52, 51, 53, 98, 97, 98, 99, 54, 101, 102, 55, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_1f4d04d5-0620-435e-8421-435babc6ef77." peer_addr = [fd00:1122:3344:101::3]:34018 req_id = 32034b51-d7d2-49e0-8775-fedc77c76d1d
752026-07-23T06:30:00.192ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51401, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:65048 req_id = 9e2b20b8-52eb-4cf6-8695-108641e8afd0
762026-07-23T06:30:00.295ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62456, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5ff6446f-a1ab-4ae6-a1ba-7a6412f38a98."), ), original: Query { name: Name("oxz_cockroachdb_5ff6446f-a1ab-4ae6-a1ba-7a6412f38a98."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 102, 102, 54, 52, 52, 54, 102, 45, 97, 49, 97, 98, 45, 52, 97, 101, 54, 45, 97, 49, 98, 97, 45, 55, 97, 54, 52, 49, 50, 102, 51, 56, 97, 57, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5ff6446f-a1ab-4ae6-a1ba-7a6412f38a98." peer_addr = [fd00:1122:3344:101::7]:33313 req_id = a4114477-38ba-42d5-b04a-7f5006a41c25
772026-07-23T06:30:00.295ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62456, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5ff6446f-a1ab-4ae6-a1ba-7a6412f38a98."), ), original: Query { name: Name("oxz_cockroachdb_5ff6446f-a1ab-4ae6-a1ba-7a6412f38a98."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 102, 102, 54, 52, 52, 54, 102, 45, 97, 49, 97, 98, 45, 52, 97, 101, 54, 45, 97, 49, 98, 97, 45, 55, 97, 54, 52, 49, 50, 102, 51, 56, 97, 57, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5ff6446f-a1ab-4ae6-a1ba-7a6412f38a98." peer_addr = [fd00:1122:3344:101::7]:38617 req_id = 71721b67-24b5-4d56-9a65-7a24a0cf3677
782026-07-23T06:30:00.301ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54734, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_86ce8f8f-4a51-4893-808f-9cbbaa7bff9a."), ), original: Query { name: Name("oxz_cockroachdb_86ce8f8f-4a51-4893-808f-9cbbaa7bff9a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 54, 99, 101, 56, 102, 56, 102, 45, 52, 97, 53, 49, 45, 52, 56, 57, 51, 45, 56, 48, 56, 102, 45, 57, 99, 98, 98, 97, 97, 55, 98, 102, 102, 57, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_86ce8f8f-4a51-4893-808f-9cbbaa7bff9a." peer_addr = [fd00:1122:3344:101::6]:49876 req_id = 3218ffbe-fa55-43e2-9c76-978c6d0cc457
792026-07-23T06:30:00.302ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54734, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_86ce8f8f-4a51-4893-808f-9cbbaa7bff9a."), ), original: Query { name: Name("oxz_cockroachdb_86ce8f8f-4a51-4893-808f-9cbbaa7bff9a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 54, 99, 101, 56, 102, 56, 102, 45, 52, 97, 53, 49, 45, 52, 56, 57, 51, 45, 56, 48, 56, 102, 45, 57, 99, 98, 98, 97, 97, 55, 98, 102, 102, 57, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_86ce8f8f-4a51-4893-808f-9cbbaa7bff9a." peer_addr = [fd00:1122:3344:101::6]:60640 req_id = 09c7337a-6dda-4c5e-8f09-e588e33276d1
802026-07-23T06:30:00.864ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28148, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:58870 req_id = fe235fd9-2f58-4a4d-97aa-348d5ba219a5
812026-07-23T06:30:00.865ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28148, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:42072 req_id = c9c783a8-e520-4a94-a8f4-0986b9f0beee
822026-07-23T06:30:01.373ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5722, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_2d9499e3-3009-41f7-9426-c043bf36137c."), ), original: Query { name: Name("oxz_ntp_2d9499e3-3009-41f7-9426-c043bf36137c."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 50, 100, 57, 52, 57, 57, 101, 51, 45, 51, 48, 48, 57, 45, 52, 49, 102, 55, 45, 57, 52, 50, 54, 45, 99, 48, 52, 51, 98, 102, 51, 54, 49, 51, 55, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_2d9499e3-3009-41f7-9426-c043bf36137c." peer_addr = [fd00:1122:3344:101::1b]:57075 req_id = a1dbac16-02be-431b-8ffb-a3b36785a2b0