Max level shown:
1 [ Jul 28 03:09:31 Enabled. ]
2 [ Jul 28 03:09:31 Rereading configuration. ]
3 [ Jul 28 03:09:31 Rereading configuration. ]
4 [ Jul 28 03:09:33 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Jul 28 03:09:33 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-28T03:09:33.174ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-07-28T03:09:33.175ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-28T03:09:33.195ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-28T03:09:33.195ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-28T03:09:33.195ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-07-28T03:09:33.196ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-07-28T03:09:33.506ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60668, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:42276 req_id = 001a9c19-3470-40fd-803a-ca9833727071
152026-07-28T03:09:33.513ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:46541
162026-07-28T03:09:33.514ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 1a2710fe-9080-4dbc-ad02-693f1d06914a
172026-07-28T03:09:33.514ZINFOdns-server (store): pruning trees for generations newer than 0
182026-07-28T03:09:33.514ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48382, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:40809 req_id = 4fff0f75-9aed-473d-924e-a886e7c83be1
192026-07-28T03:09:33.515ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31859, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("bd69b1dd-776c-498f-a1ad-4e3f287c7d83.host.control-plane.oxide.internal."), ), original: Query { name: Name("bd69b1dd-776c-498f-a1ad-4e3f287c7d83.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 98, 100, 54, 57, 98, 49, 100, 100, 45, 55, 55, 54, 99, 45, 52, 57, 56, 102, 45, 97, 49, 97, 100, 45, 52, 101, 51, 102, 50, 56, 55, 99, 55, 100, 56, 51, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "bd69b1dd-776c-498f-a1ad-4e3f287c7d83.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:41764 req_id = c2ba9a8f-493d-4b94-ab0b-f7c6444c7065
202026-07-28T03:09:33.515ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7546, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("b0ebb76c-e9a7-440f-9ccd-ca26541a0805.host.control-plane.oxide.internal."), ), original: Query { name: Name("b0ebb76c-e9a7-440f-9ccd-ca26541a0805.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 98, 48, 101, 98, 98, 55, 54, 99, 45, 101, 57, 97, 55, 45, 52, 52, 48, 102, 45, 57, 99, 99, 100, 45, 99, 97, 50, 54, 53, 52, 49, 97, 48, 56, 48, 53, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "b0ebb76c-e9a7-440f-9ccd-ca26541a0805.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:34657 req_id = bc7f07e4-8321-4136-b10d-28394f720f3f
212026-07-28T03:09:33.515ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14717, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("dca87a9a-e9f0-4227-9ca8-edb4b534193e.host.control-plane.oxide.internal."), ), original: Query { name: Name("dca87a9a-e9f0-4227-9ca8-edb4b534193e.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 100, 99, 97, 56, 55, 97, 57, 97, 45, 101, 57, 102, 48, 45, 52, 50, 50, 55, 45, 57, 99, 97, 56, 45, 101, 100, 98, 52, 98, 53, 51, 52, 49, 57, 51, 101, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "dca87a9a-e9f0-4227-9ca8-edb4b534193e.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:30748 req_id = 3a6a7e78-5fa2-4c8a-87eb-1aae3e78b82f
222026-07-28T03:09:33.536ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
232026-07-28T03:09:33.536ZINFOdns-server (store): updated generation new_generation = 1 req_id = 1a2710fe-9080-4dbc-ad02-693f1d06914a
242026-07-28T03:09:33.536ZINFOdns-server (http): request completed latency_us = 22946 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:46541 req_id = 1a2710fe-9080-4dbc-ad02-693f1d06914a response_code = 204 uri = /config
252026-07-28T03:09:46.295ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4332, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:39564 req_id = 8aa725fc-f2d9-4198-b5e6-086db4e3b610
262026-07-28T03:09:46.300ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45111, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:51379 req_id = 673653ec-e8e6-435a-8db8-9edf52ac39f8
272026-07-28T03:09:46.304ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45674, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:32887 req_id = 99dadd02-b27f-4767-aff8-a2b2ded70678
282026-07-28T03:11:10.076ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27098, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:52689 req_id = 53c371b7-6e77-4c6f-9965-e7d99c5ca430
292026-07-28T03:11:10.081ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49808 req_id = de6c067d-6951-4d57-9c86-e15c191165e8
302026-07-28T03:11:10.086ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14740, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:40727 req_id = 1f5af8ac-cc8c-473a-b4cd-738e5e596b6e
312026-07-28T03:12:52.160ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31179, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:65084 req_id = de18655b-95b9-4d76-a28b-9968851300eb
322026-07-28T03:12:54.151ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47774, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:56979 req_id = 453499e9-5d7d-4187-bf92-1cfd28f54fd9
332026-07-28T03:12:54.158ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47774, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:53959 req_id = b5f66192-3a1c-4612-a84f-ca16a6a9033d
342026-07-28T03:13:01.331ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42809, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:13599 req_id = 4f4f29a6-60e1-449e-9ffe-858ac59c20bc
352026-07-28T03:13:01.473ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1234, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:59541 req_id = bdbe0b99-7aa4-4ff9-8d91-e9cabc0ab619
362026-07-28T03:13:01.809ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60043, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:34146 req_id = f47b9677-338b-4a84-a3f9-50f15862f705
372026-07-28T03:13:02.120ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64875 req_id = fd33a6f5-4fb5-4723-a24c-4d99ddf66fa4
382026-07-28T03:13:03.099ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:41345
392026-07-28T03:13:03.150ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 279d71a3-2294-4c26-8670-db3befa85871
402026-07-28T03:13:03.150ZINFOdns-server (store): updated generation new_generation = 1 req_id = 279d71a3-2294-4c26-8670-db3befa85871
412026-07-28T03:13:03.150ZINFOdns-server (http): request completed latency_us = 24238 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:41345 req_id = 279d71a3-2294-4c26-8670-db3befa85871 response_code = 204 uri = /config
422026-07-28T03:13:03.303ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:62345
432026-07-28T03:13:03.303ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:49409
442026-07-28T03:13:03.303ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:59441
452026-07-28T03:13:03.309ZINFOdns-server (http): request completed latency_us = 5990 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:49409 req_id = 30436491-159e-49bd-801c-3c0e1493339d response_code = 200 uri = /config
462026-07-28T03:13:03.309ZINFOdns-server (http): request completed latency_us = 6008 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:62345 req_id = 27ab213c-d4ee-4f20-b357-33ac94f386dc response_code = 200 uri = /config
472026-07-28T03:13:03.309ZINFOdns-server (http): request completed latency_us = 5998 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:59441 req_id = 634e1db1-fa8f-4eb9-ae5f-ee49f9e3f72a response_code = 200 uri = /config
482026-07-28T03:13:04.253ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33277, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:48656 req_id = 1117e0de-e7e1-4d3e-8ed0-b162ee539054
492026-07-28T03:13:04.253ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21189, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:39326 req_id = 1aa54d30-04fd-4074-96ef-941ff0d91251
502026-07-28T03:13:04.389ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:40124
512026-07-28T03:13:04.389ZINFOdns-server (http): request completed latency_us = 209 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:40124 req_id = fb759207-1a8d-42de-822f-f1efea12682c response_code = 200 uri = /config
522026-07-28T03:13:04.507ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:39805
532026-07-28T03:13:04.507ZINFOdns-server (http): request completed latency_us = 300 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:39805 req_id = f6f94d02-cc04-4357-809a-c7026c4ee9b3 response_code = 200 uri = /config
542026-07-28T03:13:05.385ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34895, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:42769 req_id = 478a1f5e-9829-498c-816e-994f579a77be
552026-07-28T03:13:05.607ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:39165
562026-07-28T03:13:05.608ZINFOdns-server (http): request completed latency_us = 232 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:39165 req_id = 785e9c49-6960-4d26-8106-03419e71d97a response_code = 200 uri = /config
572026-07-28T03:13:12.122ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26758, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62268 req_id = 7b4c7d05-55c7-411b-aac4-aa425980c401
582026-07-28T03:13:12.382ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10179, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:44367 req_id = a54ecb70-26f2-40f4-900a-5541db775e37
592026-07-28T03:13:12.387ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35756, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:34259 req_id = a7441392-e5b6-4670-a1ea-fa9b13fb3930
602026-07-28T03:13:12.509ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:33128
612026-07-28T03:13:12.510ZINFOdns-server (http): request completed latency_us = 214 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:33128 req_id = abf82bdb-e089-4551-9921-ed1ce76fd7d9 response_code = 200 uri = /config
622026-07-28T03:13:12.579ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:33931
632026-07-28T03:13:12.579ZINFOdns-server (http): request completed latency_us = 190 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:33931 req_id = ab1fc564-e146-4e7f-af69-235c53f55d1a response_code = 200 uri = /config
642026-07-28T03:13:22.123ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1578, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62200 req_id = 73111549-43f7-48b7-a468-6f76cd014dd2
652026-07-28T03:13:32.125ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20474, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50675 req_id = f06bcc4e-91e9-424e-b2f3-c301e6ac7b23
662026-07-28T03:13:38.286ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32861, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:34098 req_id = cd9f8b90-e2a4-42f4-9ad7-297352b81f32
672026-07-28T03:13:38.290ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62660, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49274 req_id = 1206873a-80bb-40cf-a3cb-68e247f36a33
682026-07-28T03:13:38.294ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18405, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49745 req_id = 7c97b470-8cfc-4e2c-a31c-e195c8396aad
692026-07-28T03:13:42.126ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45518, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63314 req_id = 2cd8747d-49fe-48c6-9843-1ea99eab0d39
702026-07-28T03:13:52.128ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53214, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:65289 req_id = c60f4536-92ed-4a0b-890b-5eec7c41b9c8
712026-07-28T03:14:01.289ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:33868
722026-07-28T03:14:01.291ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 862ecc79-34c4-4112-826b-f0fc73ceb77f
732026-07-28T03:14:01.291ZINFOdns-server (store): updated generation new_generation = 1 req_id = 862ecc79-34c4-4112-826b-f0fc73ceb77f
742026-07-28T03:14:01.291ZINFOdns-server (http): request completed latency_us = 323 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:33868 req_id = 862ecc79-34c4-4112-826b-f0fc73ceb77f response_code = 204 uri = /config
752026-07-28T03:14:01.456ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:51446
762026-07-28T03:14:01.458ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 57d4e081-983b-4049-a96d-58a8fb11e6fe
772026-07-28T03:14:01.458ZINFOdns-server (store): updated generation new_generation = 1 req_id = 57d4e081-983b-4049-a96d-58a8fb11e6fe
782026-07-28T03:14:01.458ZINFOdns-server (http): request completed latency_us = 280 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:51446 req_id = 57d4e081-983b-4049-a96d-58a8fb11e6fe response_code = 204 uri = /config
792026-07-28T03:14:01.711ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:40631
802026-07-28T03:14:01.713ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = f18a4d60-26e7-463d-aa13-ef2f79d46535
812026-07-28T03:14:01.713ZINFOdns-server (store): updated generation new_generation = 1 req_id = f18a4d60-26e7-463d-aa13-ef2f79d46535
822026-07-28T03:14:01.713ZINFOdns-server (http): request completed latency_us = 259 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:40631 req_id = f18a4d60-26e7-463d-aa13-ef2f79d46535 response_code = 204 uri = /config
832026-07-28T03:14:02.130ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24487, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54483 req_id = ae052858-c607-421a-ba8c-b5c613c0e039
842026-07-28T03:14:03.055ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7568, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:51818 req_id = 217ee81e-bc73-41a5-9c1f-e4b33c095392
852026-07-28T03:14:03.242ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:63438
862026-07-28T03:14:03.242ZINFOdns-server (http): request completed latency_us = 186 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:63438 req_id = b75688d6-ac9c-417e-92ab-636dcd8872f6 response_code = 200 uri = /config
872026-07-28T03:14:03.607ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7049, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:33801 req_id = 6ce53c65-ee4b-4124-b478-a894ef48e4a8
882026-07-28T03:14:03.706ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:49301
892026-07-28T03:14:03.706ZINFOdns-server (http): request completed latency_us = 185 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:49301 req_id = dead0aa6-6693-421d-ba30-35e7aaf061a1 response_code = 200 uri = /config
902026-07-28T03:14:04.152ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46321, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:51570 req_id = cecc2f86-9b83-412f-b964-cdea52ed4b4e
912026-07-28T03:14:04.248ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:33289
922026-07-28T03:14:04.248ZINFOdns-server (http): request completed latency_us = 193 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:33289 req_id = 02ae2592-be46-464a-be01-dc984f9ae182 response_code = 200 uri = /config
932026-07-28T03:14:12.132ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44810, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58777 req_id = 9a5958ee-6054-4d92-bb3a-d13eea9dce61
942026-07-28T03:14:22.134ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51799, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50794 req_id = c2a905bb-ed5f-4048-89ea-e0ed92cecd7b
952026-07-28T03:14:32.136ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19603, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60001 req_id = b4065826-8d66-488f-a35d-69d9e23503f7
962026-07-28T03:14:42.138ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52989, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55257 req_id = 18b3a4df-31bf-4b60-899b-21b1e23edd3b
972026-07-28T03:14:52.140ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48696, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58348 req_id = 387f6bc5-f30d-431f-953a-e8f97fbf4ea7
982026-07-28T03:15:00.435ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21997, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), ), original: Query { name: Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 55, 52, 52, 50, 57, 98, 100, 45, 50, 102, 101, 48, 45, 52, 101, 99, 52, 45, 57, 101, 102, 57, 45, 53, 102, 55, 98, 98, 52, 49, 51, 102, 57, 51, 49, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931." peer_addr = [fd00:1122:3344:101::3]:51150 req_id = 4bb17b4d-deaa-4e48-b547-aed6e3b76f10
992026-07-28T03:15:00.436ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21997, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), ), original: Query { name: Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 55, 52, 52, 50, 57, 98, 100, 45, 50, 102, 101, 48, 45, 52, 101, 99, 52, 45, 57, 101, 102, 57, 45, 53, 102, 55, 98, 98, 52, 49, 51, 102, 57, 51, 49, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931." peer_addr = [fd00:1122:3344:101::3]:58667 req_id = 8b62a28a-81cb-4f08-b9d2-452bc1de3e7e
1002026-07-28T03:15:00.459ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56693, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:44584 req_id = 43f63f60-a134-4354-b723-6006eecb772e
1012026-07-28T03:15:00.460ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56693, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:33446 req_id = d5952dcd-9b78-4472-a435-00ce08f4abc8
1022026-07-28T03:15:00.478ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59429, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), ), original: Query { name: Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 53, 101, 101, 98, 100, 101, 55, 45, 53, 98, 49, 51, 45, 52, 98, 55, 52, 45, 97, 52, 56, 97, 45, 54, 97, 57, 48, 53, 49, 100, 99, 51, 98, 100, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4." peer_addr = [fd00:1122:3344:101::6]:60746 req_id = af8b418e-4455-4f0c-b019-6eec43387326
1032026-07-28T03:15:00.478ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59429, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), ), original: Query { name: Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 53, 101, 101, 98, 100, 101, 55, 45, 53, 98, 49, 51, 45, 52, 98, 55, 52, 45, 97, 52, 56, 97, 45, 54, 97, 57, 48, 53, 49, 100, 99, 51, 98, 100, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4." peer_addr = [fd00:1122:3344:101::6]:48318 req_id = 925154a6-100f-4957-9a44-b850360fd1dd
1042026-07-28T03:15:00.543ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45212, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), ), original: Query { name: Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 97, 100, 57, 101, 49, 102, 55, 45, 53, 57, 102, 52, 45, 52, 57, 48, 54, 45, 98, 99, 97, 50, 45, 97, 102, 54, 56, 100, 53, 56, 101, 99, 55, 54, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764." peer_addr = [fd00:1122:3344:101::4]:51120 req_id = 4ed7c1eb-6dc6-40fa-81d5-af3d4db100d5
1052026-07-28T03:15:00.544ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45212, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), ), original: Query { name: Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 97, 100, 57, 101, 49, 102, 55, 45, 53, 57, 102, 52, 45, 52, 57, 48, 54, 45, 98, 99, 97, 50, 45, 97, 102, 54, 56, 100, 53, 56, 101, 99, 55, 54, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764." peer_addr = [fd00:1122:3344:101::4]:35155 req_id = eb958c1b-cf4f-457b-939f-15d94b22a85a
1062026-07-28T03:15:00.592ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5920, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), ), original: Query { name: Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 97, 57, 99, 49, 49, 99, 49, 45, 55, 55, 48, 49, 45, 52, 101, 54, 99, 45, 98, 55, 55, 48, 45, 101, 53, 101, 100, 97, 102, 48, 101, 48, 54, 54, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668." peer_addr = [fd00:1122:3344:101::7]:46457 req_id = 13c66cf9-f5a2-47ab-846e-58ccc0b2bbfd
1072026-07-28T03:15:00.593ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5920, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), ), original: Query { name: Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 97, 57, 99, 49, 49, 99, 49, 45, 55, 55, 48, 49, 45, 52, 101, 54, 99, 45, 98, 55, 55, 48, 45, 101, 53, 101, 100, 97, 102, 48, 101, 48, 54, 54, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668." peer_addr = [fd00:1122:3344:101::7]:53473 req_id = 341a8799-3f9d-46b0-9ec4-e6acb1dd4978
1082026-07-28T03:15:00.626ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52933, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), ), original: Query { name: Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 97, 53, 54, 99, 101, 99, 99, 45, 49, 57, 52, 102, 45, 52, 97, 99, 55, 45, 97, 56, 50, 54, 45, 49, 52, 52, 56, 97, 100, 53, 49, 53, 98, 51, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32." peer_addr = [fd00:1122:3344:101::5]:39043 req_id = a620e20c-9d2b-4278-8a9d-ba7038ee7a2f
1092026-07-28T03:15:00.627ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52933, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), ), original: Query { name: Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 97, 53, 54, 99, 101, 99, 99, 45, 49, 57, 52, 102, 45, 52, 97, 99, 55, 45, 97, 56, 50, 54, 45, 49, 52, 52, 56, 97, 100, 53, 49, 53, 98, 51, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32." peer_addr = [fd00:1122:3344:101::5]:56709 req_id = 2fb60b77-f499-4a38-896a-ff1cf57d1a58
1102026-07-28T03:15:00.888ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:40200 req_id = 49c54f3e-e591-4462-b5d1-3f93c80bf386
1112026-07-28T03:15:00.889ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:39162 req_id = 8e474572-fc96-442d-8b20-6f8322359f18
1122026-07-28T03:15:01.112ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:36693
1132026-07-28T03:15:01.114ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = c3c26110-bf3f-4db7-8527-4bf3d1c39b67
1142026-07-28T03:15:01.114ZINFOdns-server (store): updated generation new_generation = 1 req_id = c3c26110-bf3f-4db7-8527-4bf3d1c39b67
1152026-07-28T03:15:01.114ZINFOdns-server (http): request completed latency_us = 263 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:36693 req_id = c3c26110-bf3f-4db7-8527-4bf3d1c39b67 response_code = 204 uri = /config
1162026-07-28T03:15:01.121ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35828, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_fa5d5421-7e64-424d-97f0-785be1f87393."), ), original: Query { name: Name("oxz_ntp_fa5d5421-7e64-424d-97f0-785be1f87393."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 102, 97, 53, 100, 53, 52, 50, 49, 45, 55, 101, 54, 52, 45, 52, 50, 52, 100, 45, 57, 55, 102, 48, 45, 55, 56, 53, 98, 101, 49, 102, 56, 55, 51, 57, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_fa5d5421-7e64-424d-97f0-785be1f87393." peer_addr = [fd00:1122:3344:101::1b]:62335 req_id = 120b4e80-996c-4528-97da-235869197314
1172026-07-28T03:15:01.181ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:48717
1182026-07-28T03:15:01.183ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 2ca06b0c-4f2a-4ee6-9716-6f26f5efc7ce
1192026-07-28T03:15:01.183ZINFOdns-server (store): updated generation new_generation = 1 req_id = 2ca06b0c-4f2a-4ee6-9716-6f26f5efc7ce
1202026-07-28T03:15:01.183ZINFOdns-server (http): request completed latency_us = 282 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:48717 req_id = 2ca06b0c-4f2a-4ee6-9716-6f26f5efc7ce response_code = 204 uri = /config