Max level shown:
1 [ Jul 28 03:09:31 Enabled. ]
2 [ Jul 28 03:09:31 Rereading configuration. ]
3 [ Jul 28 03:09:32 Rereading configuration. ]
4 [ Jul 28 03:09:33 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Jul 28 03:09:33 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-28T03:09:33.222ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-07-28T03:09:33.222ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-28T03:09:33.240ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-28T03:09:33.240ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-28T03:09:33.240ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-07-28T03:09:33.241ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-07-28T03:09:33.506ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24893, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:57836 req_id = da4a997d-2a9e-4989-a732-bec5d077a610
152026-07-28T03:09:33.514ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38336, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:17859 req_id = 6853906c-cb06-4822-ae88-1ae61d776de5
162026-07-28T03:09:33.515ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48707, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("bd69b1dd-776c-498f-a1ad-4e3f287c7d83.host.control-plane.oxide.internal."), ), original: Query { name: Name("bd69b1dd-776c-498f-a1ad-4e3f287c7d83.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 98, 100, 54, 57, 98, 49, 100, 100, 45, 55, 55, 54, 99, 45, 52, 57, 56, 102, 45, 97, 49, 97, 100, 45, 52, 101, 51, 102, 50, 56, 55, 99, 55, 100, 56, 51, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "bd69b1dd-776c-498f-a1ad-4e3f287c7d83.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:19701 req_id = 8aca08c6-e3ff-458f-a5f1-87b7c6d72f48
172026-07-28T03:09:33.515ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46094, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("b0ebb76c-e9a7-440f-9ccd-ca26541a0805.host.control-plane.oxide.internal."), ), original: Query { name: Name("b0ebb76c-e9a7-440f-9ccd-ca26541a0805.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 98, 48, 101, 98, 98, 55, 54, 99, 45, 101, 57, 97, 55, 45, 52, 52, 48, 102, 45, 57, 99, 99, 100, 45, 99, 97, 50, 54, 53, 52, 49, 97, 48, 56, 48, 53, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "b0ebb76c-e9a7-440f-9ccd-ca26541a0805.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:21501 req_id = 6a2be5d4-292c-4c01-b6c8-59ec8efb4c82
182026-07-28T03:09:33.515ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9956, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("dca87a9a-e9f0-4227-9ca8-edb4b534193e.host.control-plane.oxide.internal."), ), original: Query { name: Name("dca87a9a-e9f0-4227-9ca8-edb4b534193e.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 100, 99, 97, 56, 55, 97, 57, 97, 45, 101, 57, 102, 48, 45, 52, 50, 50, 55, 45, 57, 99, 97, 56, 45, 101, 100, 98, 52, 98, 53, 51, 52, 49, 57, 51, 101, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "dca87a9a-e9f0-4227-9ca8-edb4b534193e.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:38158 req_id = b30f67aa-86dd-4cd6-ae0d-a1b2052b389e
192026-07-28T03:09:33.538ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:61666
202026-07-28T03:09:33.539ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 71b8be32-2345-45b9-a2c7-e04f07cb2634
212026-07-28T03:09:33.539ZINFOdns-server (store): pruning trees for generations newer than 0
222026-07-28T03:09:33.562ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
232026-07-28T03:09:33.562ZINFOdns-server (store): updated generation new_generation = 1 req_id = 71b8be32-2345-45b9-a2c7-e04f07cb2634
242026-07-28T03:09:33.562ZINFOdns-server (http): request completed latency_us = 23320 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:61666 req_id = 71b8be32-2345-45b9-a2c7-e04f07cb2634 response_code = 204 uri = /config
252026-07-28T03:09:46.296ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4332, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:45495 req_id = 18857d63-7910-42ca-92a5-38adee8a76bf
262026-07-28T03:09:46.300ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45111, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:50499 req_id = f5ae3edb-54d5-451c-ac2d-98c7ffabfde5
272026-07-28T03:09:46.305ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45674, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:54933 req_id = 67536a65-e2cd-47d7-b319-315d2c44e2cb
282026-07-28T03:11:10.077ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27098, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:43031 req_id = 43054c82-164d-49f0-8135-b230872d61d5
292026-07-28T03:11:10.081ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:40905 req_id = 91ff414b-ac8b-4ec7-9e8a-1ce9d03a5449
302026-07-28T03:11:10.086ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14740, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:39294 req_id = b79c91d4-9779-4b23-b980-8cb518a8005f
312026-07-28T03:12:52.158ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64847, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57245 req_id = 0bc8ef28-8a68-4833-864c-5c1dfced2e2d
322026-07-28T03:12:54.157ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47774, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:41798 req_id = 260deaae-4184-4e6d-a3c0-11fe5ba88339
332026-07-28T03:12:54.158ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47774, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:44301 req_id = 3fcddf02-8460-4ee4-a706-bb335d8815ac
342026-07-28T03:13:01.331ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4697, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:32880 req_id = 56efbb3e-21fd-4cce-8629-e168727203f2
352026-07-28T03:13:01.473ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12311, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:23340 req_id = 3c1588e2-10a4-4ea1-9ece-edd85e0dde33
362026-07-28T03:13:01.809ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43445, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:3225 req_id = 6b4414f3-b4fc-4593-b314-aa74b8154ea0
372026-07-28T03:13:02.120ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6588, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52924 req_id = 8f71e435-a208-4935-87a8-e9e3bfe38327
382026-07-28T03:13:03.098ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:45642
392026-07-28T03:13:03.150ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 1a9b7f70-46c1-4113-92b6-b70e3ade5171
402026-07-28T03:13:03.150ZINFOdns-server (store): updated generation new_generation = 1 req_id = 1a9b7f70-46c1-4113-92b6-b70e3ade5171
412026-07-28T03:13:03.150ZINFOdns-server (http): request completed latency_us = 25183 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:45642 req_id = 1a9b7f70-46c1-4113-92b6-b70e3ade5171 response_code = 204 uri = /config
422026-07-28T03:13:03.310ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:37646
432026-07-28T03:13:03.310ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:37147
442026-07-28T03:13:03.310ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:56259
452026-07-28T03:13:03.317ZINFOdns-server (http): request completed latency_us = 6919 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:56259 req_id = 9a2b51b2-4cf8-4f57-bcb1-9e14ce69ca38 response_code = 200 uri = /config
462026-07-28T03:13:03.317ZINFOdns-server (http): request completed latency_us = 6923 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:37646 req_id = eb995776-cca5-4db8-8b42-8106d64b7045 response_code = 200 uri = /config
472026-07-28T03:13:03.317ZINFOdns-server (http): request completed latency_us = 6947 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:37147 req_id = 6f900838-b6fa-4ea8-ae70-f2cb57dacab6 response_code = 200 uri = /config
482026-07-28T03:13:04.253ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44008, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:60512 req_id = bf904bcf-59ff-4b9b-b2e9-acea066c48d3
492026-07-28T03:13:04.253ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49033, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:20927 req_id = b8c2c428-b33a-4ce5-a4dd-3ef6d74855f2
502026-07-28T03:13:04.390ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:38261
512026-07-28T03:13:04.390ZINFOdns-server (http): request completed latency_us = 224 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:38261 req_id = 2face485-3c91-4716-a08a-56dc1ca6586f response_code = 200 uri = /config
522026-07-28T03:13:04.508ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:49946
532026-07-28T03:13:04.508ZINFOdns-server (http): request completed latency_us = 237 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:49946 req_id = 017c4dbf-6f65-4355-b74c-a8d3ab2c16ab response_code = 200 uri = /config
542026-07-28T03:13:05.385ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1127, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:58177 req_id = 48ad1976-1082-4394-866d-18a7fb88e216
552026-07-28T03:13:05.608ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:42981
562026-07-28T03:13:05.609ZINFOdns-server (http): request completed latency_us = 195 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:42981 req_id = 28f05a91-4bd2-4277-b8a4-1e913f4e99d3 response_code = 200 uri = /config
572026-07-28T03:13:12.122ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40645, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52288 req_id = 9a9100e2-5c44-4c1b-b567-2f5594e5ca54
582026-07-28T03:13:12.382ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47565, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:49257 req_id = 6e1f6b97-e98e-40f1-bd5a-691084b72158
592026-07-28T03:13:12.387ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4782, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:45469 req_id = b5d4f762-59d7-46da-a337-5fd10f06e41d
602026-07-28T03:13:12.510ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:64926
612026-07-28T03:13:12.510ZINFOdns-server (http): request completed latency_us = 242 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:64926 req_id = 85d98e38-dbd6-47d7-a171-cb052e973593 response_code = 200 uri = /config
622026-07-28T03:13:12.580ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:62465
632026-07-28T03:13:12.580ZINFOdns-server (http): request completed latency_us = 198 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:62465 req_id = d40562d6-9883-4e41-accd-d5ec46939a38 response_code = 200 uri = /config
642026-07-28T03:13:22.123ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52462, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57452 req_id = dc023111-6759-455e-9f72-c19f8dcead90
652026-07-28T03:13:32.125ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12592, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64208 req_id = 0461cd26-30ef-4c46-82ec-dcba80a357f2
662026-07-28T03:13:38.286ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32861, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:62632 req_id = cfa556ac-5d29-4884-a0ff-c3558fa78cf5
672026-07-28T03:13:38.290ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62660, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:35784 req_id = fcd4fe21-bece-4212-8049-68da8d086815
682026-07-28T03:13:38.294ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18405, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:46621 req_id = 9e513483-a93a-421d-a005-f966590ae493
692026-07-28T03:13:42.126ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 221, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54494 req_id = 1671cadf-53d2-49f8-bd3a-ba8743623acf
702026-07-28T03:13:52.128ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15586, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54905 req_id = 4650787a-289c-47ba-b98d-a6ac852d17df
712026-07-28T03:14:01.290ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:51764
722026-07-28T03:14:01.291ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 83d14ba3-7be6-46e7-bbbc-4aac2325e9e4
732026-07-28T03:14:01.291ZINFOdns-server (store): updated generation new_generation = 1 req_id = 83d14ba3-7be6-46e7-bbbc-4aac2325e9e4
742026-07-28T03:14:01.291ZINFOdns-server (http): request completed latency_us = 258 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:51764 req_id = 83d14ba3-7be6-46e7-bbbc-4aac2325e9e4 response_code = 204 uri = /config
752026-07-28T03:14:01.458ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:63636
762026-07-28T03:14:01.458ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 63473554-6ef2-4d6a-9274-54bc3ea0ab58
772026-07-28T03:14:01.458ZINFOdns-server (store): updated generation new_generation = 1 req_id = 63473554-6ef2-4d6a-9274-54bc3ea0ab58
782026-07-28T03:14:01.458ZINFOdns-server (http): request completed latency_us = 272 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:63636 req_id = 63473554-6ef2-4d6a-9274-54bc3ea0ab58 response_code = 204 uri = /config
792026-07-28T03:14:01.713ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:50734
802026-07-28T03:14:01.713ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 09fbdcc1-8eff-48cc-a5ff-5d1f440b5319
812026-07-28T03:14:01.713ZINFOdns-server (store): updated generation new_generation = 1 req_id = 09fbdcc1-8eff-48cc-a5ff-5d1f440b5319
822026-07-28T03:14:01.713ZINFOdns-server (http): request completed latency_us = 232 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:50734 req_id = 09fbdcc1-8eff-48cc-a5ff-5d1f440b5319 response_code = 204 uri = /config
832026-07-28T03:14:02.130ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24606, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51316 req_id = 2c6c5be0-e997-4e64-bcd8-cea94bbc2214
842026-07-28T03:14:03.055ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20337, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:4840 req_id = 3cb97331-4e4e-4493-8f14-74856232789b
852026-07-28T03:14:03.242ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:44444
862026-07-28T03:14:03.242ZINFOdns-server (http): request completed latency_us = 184 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:44444 req_id = 517f3d35-c226-41f1-a8c0-7548d3b47397 response_code = 200 uri = /config
872026-07-28T03:14:03.607ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52097, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:25669 req_id = 2f2efa0d-81f2-4001-b4a8-1f20069dc942
882026-07-28T03:14:03.707ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:40790
892026-07-28T03:14:03.707ZINFOdns-server (http): request completed latency_us = 203 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:40790 req_id = b8712617-142d-49c4-a897-34f50499072e response_code = 200 uri = /config
902026-07-28T03:14:04.152ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61100, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:11019 req_id = ab3b06c3-203c-4a18-9d6f-14180f4b2ccc
912026-07-28T03:14:04.249ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:33930
922026-07-28T03:14:04.249ZINFOdns-server (http): request completed latency_us = 186 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:33930 req_id = cad0cdd1-b816-433c-acab-a90dc147632e response_code = 200 uri = /config
932026-07-28T03:14:12.132ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24449, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58730 req_id = d1290b02-4980-4c1a-86c9-6abda73f9594
942026-07-28T03:14:22.134ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4432, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64455 req_id = f6e13d40-7960-4ea9-a70b-efbff413e7d4
952026-07-28T03:14:32.136ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3103, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55999 req_id = 9e778b06-ae18-4cc0-b0f0-f912d97e99ea
962026-07-28T03:14:42.138ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57852 req_id = 6f798388-fd53-4f7c-a8b2-859e1a1d06bd
972026-07-28T03:14:52.140ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34852, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52563 req_id = 4f6b2a7c-ff02-4bb6-82ca-587149e470e9
982026-07-28T03:15:00.436ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21997, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), ), original: Query { name: Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 55, 52, 52, 50, 57, 98, 100, 45, 50, 102, 101, 48, 45, 52, 101, 99, 52, 45, 57, 101, 102, 57, 45, 53, 102, 55, 98, 98, 52, 49, 51, 102, 57, 51, 49, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931." peer_addr = [fd00:1122:3344:101::3]:37687 req_id = 917a1af1-d227-4590-9ce7-0e1935fcab42
992026-07-28T03:15:00.436ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21997, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), ), original: Query { name: Name("oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 55, 52, 52, 50, 57, 98, 100, 45, 50, 102, 101, 48, 45, 52, 101, 99, 52, 45, 57, 101, 102, 57, 45, 53, 102, 55, 98, 98, 52, 49, 51, 102, 57, 51, 49, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_174429bd-2fe0-4ec4-9ef9-5f7bb413f931." peer_addr = [fd00:1122:3344:101::3]:37461 req_id = ea942c97-85c0-4b5f-8e65-40f8b38da4bf
1002026-07-28T03:15:00.460ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56693, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:47666 req_id = 72b431f9-d239-4f26-b6fd-146ca1aab93d
1012026-07-28T03:15:00.460ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56693, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), ), original: Query { name: Name("oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 50, 55, 100, 49, 52, 54, 48, 54, 45, 53, 101, 97, 55, 45, 52, 55, 55, 50, 45, 98, 48, 99, 55, 45, 101, 102, 55, 48, 101, 56, 51, 49, 52, 50, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_27d14606-5ea7-4772-b0c7-ef70e831422a." peer_addr = [fd00:1122:3344:101::e]:61068 req_id = 437ca406-4419-4ca4-bf33-29483e697cd0
1022026-07-28T03:15:00.478ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59429, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), ), original: Query { name: Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 53, 101, 101, 98, 100, 101, 55, 45, 53, 98, 49, 51, 45, 52, 98, 55, 52, 45, 97, 52, 56, 97, 45, 54, 97, 57, 48, 53, 49, 100, 99, 51, 98, 100, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4." peer_addr = [fd00:1122:3344:101::6]:58430 req_id = 66a8ea0f-b797-4077-a283-2f3554214b16
1032026-07-28T03:15:00.478ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59429, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), ), original: Query { name: Name("oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 53, 101, 101, 98, 100, 101, 55, 45, 53, 98, 49, 51, 45, 52, 98, 55, 52, 45, 97, 52, 56, 97, 45, 54, 97, 57, 48, 53, 49, 100, 99, 51, 98, 100, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c5eebde7-5b13-4b74-a48a-6a9051dc3bd4." peer_addr = [fd00:1122:3344:101::6]:61747 req_id = f4872ddf-9d8a-4830-bc51-68dcdfed00c1
1042026-07-28T03:15:00.543ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45212, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), ), original: Query { name: Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 97, 100, 57, 101, 49, 102, 55, 45, 53, 57, 102, 52, 45, 52, 57, 48, 54, 45, 98, 99, 97, 50, 45, 97, 102, 54, 56, 100, 53, 56, 101, 99, 55, 54, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764." peer_addr = [fd00:1122:3344:101::4]:46163 req_id = b57570c6-9127-4958-9b4f-691b55c60a5e
1052026-07-28T03:15:00.544ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45212, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), ), original: Query { name: Name("oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 97, 100, 57, 101, 49, 102, 55, 45, 53, 57, 102, 52, 45, 52, 57, 48, 54, 45, 98, 99, 97, 50, 45, 97, 102, 54, 56, 100, 53, 56, 101, 99, 55, 54, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2ad9e1f7-59f4-4906-bca2-af68d58ec764." peer_addr = [fd00:1122:3344:101::4]:59500 req_id = a1acdfd1-e34e-4e82-94b3-d39286771757
1062026-07-28T03:15:00.592ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5920, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), ), original: Query { name: Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 97, 57, 99, 49, 49, 99, 49, 45, 55, 55, 48, 49, 45, 52, 101, 54, 99, 45, 98, 55, 55, 48, 45, 101, 53, 101, 100, 97, 102, 48, 101, 48, 54, 54, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668." peer_addr = [fd00:1122:3344:101::7]:42487 req_id = bce546cf-24ca-4278-b7bf-b36011e945ac
1072026-07-28T03:15:00.593ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5920, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), ), original: Query { name: Name("oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 97, 57, 99, 49, 49, 99, 49, 45, 55, 55, 48, 49, 45, 52, 101, 54, 99, 45, 98, 55, 55, 48, 45, 101, 53, 101, 100, 97, 102, 48, 101, 48, 54, 54, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5a9c11c1-7701-4e6c-b770-e5edaf0e0668." peer_addr = [fd00:1122:3344:101::7]:33270 req_id = a3297569-d22b-4686-b491-60933af1ea82
1082026-07-28T03:15:00.626ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52933, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), ), original: Query { name: Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 97, 53, 54, 99, 101, 99, 99, 45, 49, 57, 52, 102, 45, 52, 97, 99, 55, 45, 97, 56, 50, 54, 45, 49, 52, 52, 56, 97, 100, 53, 49, 53, 98, 51, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32." peer_addr = [fd00:1122:3344:101::5]:36810 req_id = 19ff20c4-b87a-4c6e-a551-c4754620f019
1092026-07-28T03:15:00.627ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52933, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), ), original: Query { name: Name("oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 97, 53, 54, 99, 101, 99, 99, 45, 49, 57, 52, 102, 45, 52, 97, 99, 55, 45, 97, 56, 50, 54, 45, 49, 52, 52, 56, 97, 100, 53, 49, 53, 98, 51, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ea56cecc-194f-4ac7-a826-1448ad515b32." peer_addr = [fd00:1122:3344:101::5]:38828 req_id = d610e581-a1c4-4910-a0df-4f7cfe3410a3
1102026-07-28T03:15:00.888ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:54972 req_id = f00310c1-d641-4468-bf95-b9802e666a70
1112026-07-28T03:15:00.889ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:41509 req_id = f75ef0ce-80aa-4714-9f1a-53858528dc64
1122026-07-28T03:15:01.114ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:59443
1132026-07-28T03:15:01.115ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = ca1c009b-acd3-4ef2-ba27-77c83ecee20e
1142026-07-28T03:15:01.115ZINFOdns-server (store): updated generation new_generation = 1 req_id = ca1c009b-acd3-4ef2-ba27-77c83ecee20e
1152026-07-28T03:15:01.115ZINFOdns-server (http): request completed latency_us = 867 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:59443 req_id = ca1c009b-acd3-4ef2-ba27-77c83ecee20e response_code = 204 uri = /config
1162026-07-28T03:15:01.121ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35828, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_fa5d5421-7e64-424d-97f0-785be1f87393."), ), original: Query { name: Name("oxz_ntp_fa5d5421-7e64-424d-97f0-785be1f87393."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 102, 97, 53, 100, 53, 52, 50, 49, 45, 55, 101, 54, 52, 45, 52, 50, 52, 100, 45, 57, 55, 102, 48, 45, 55, 56, 53, 98, 101, 49, 102, 56, 55, 51, 57, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_fa5d5421-7e64-424d-97f0-785be1f87393." peer_addr = [fd00:1122:3344:101::1b]:58449 req_id = 7a8adac0-0204-47fd-8b6a-20b22bdd932d
1172026-07-28T03:15:01.182ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:53923
1182026-07-28T03:15:01.182ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = fc0dc019-8e18-46b7-83ca-f0f1b3c7dfff
1192026-07-28T03:15:01.182ZINFOdns-server (store): updated generation new_generation = 1 req_id = fc0dc019-8e18-46b7-83ca-f0f1b3c7dfff
1202026-07-28T03:15:01.183ZINFOdns-server (http): request completed latency_us = 277 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:53923 req_id = fc0dc019-8e18-46b7-83ca-f0f1b3c7dfff response_code = 204 uri = /config