Max level shown:
1 [ May 15 02:41:28 Enabled. ]
2 [ May 15 02:41:28 Rereading configuration. ]
3 [ May 15 02:41:29 Rereading configuration. ]
4 [ May 15 02:41:30 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ May 15 02:41:30 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-05-15T02:41:30.524ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-05-15T02:41:30.524ZINFOdns-server (store): opening sled database path = /data/dns
102026-05-15T02:41:30.549ZINFOdns-server (store): pruning trees for generations newer than 0
112026-05-15T02:41:30.549ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-05-15T02:41:30.549ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-05-15T02:41:30.549ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-05-15T02:41:30.586ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 439, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:65514 req_id = e8f2d74b-853e-4668-b95b-15b8977ded8c
152026-05-15T02:41:32.213ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18568, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:3::2]:48631 req_id = cae36fec-7cab-4e21-b6b1-3487979152fd
162026-05-15T02:41:33.181ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:58367
172026-05-15T02:41:33.182ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 52115347-9d87-40f9-a4da-172db40a95e7
182026-05-15T02:41:33.182ZINFOdns-server (store): pruning trees for generations newer than 0
192026-05-15T02:41:33.199ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
202026-05-15T02:41:33.199ZINFOdns-server (store): updated generation new_generation = 1 req_id = 52115347-9d87-40f9-a4da-172db40a95e7
212026-05-15T02:41:33.199ZINFOdns-server (http): request completed latency_us = 17385 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:58367 req_id = 52115347-9d87-40f9-a4da-172db40a95e7 response_code = 204 uri = /config
222026-05-15T02:41:44.807ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20980, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:34162 req_id = 8fc78402-acc9-4443-8909-9382be0b6a02
232026-05-15T02:41:44.811ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30838, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:33157 req_id = 86ec2572-2147-4a6e-9701-34ef65c49b33
242026-05-15T02:41:44.816ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51980, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:48532 req_id = 39bc7f42-f5ee-4826-b0fa-1799dd73ab45
252026-05-15T02:42:40.661ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38040, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49475 req_id = 3541f65a-106f-4972-9b62-3731cb748acc
262026-05-15T02:42:45.739ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46370, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), ), original: Query { name: Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 50, 49, 51, 50, 102, 101, 100, 45, 97, 54, 100, 97, 45, 52, 51, 101, 57, 45, 98, 99, 98, 48, 45, 49, 55, 102, 55, 97, 99, 55, 48, 48, 51, 48, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304." peer_addr = [fd00:1122:3344:101::e]:58587 req_id = f9621e1c-f673-4c98-8733-d079bc54ca56
272026-05-15T02:42:45.739ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46370, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), ), original: Query { name: Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 50, 49, 51, 50, 102, 101, 100, 45, 97, 54, 100, 97, 45, 52, 51, 101, 57, 45, 98, 99, 98, 48, 45, 49, 55, 102, 55, 97, 99, 55, 48, 48, 51, 48, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304." peer_addr = [fd00:1122:3344:101::e]:46117 req_id = 50639e5f-bf67-4988-a3c8-22d4afecdfb8
282026-05-15T02:42:50.636ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14963, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64710 req_id = 8b2fbfc6-f3fc-4fe8-950b-7bb1a3fbbdd9
292026-05-15T02:42:52.177ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23560, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:2864 req_id = b1a09b8a-86f3-46ad-b54b-3069566cf383
302026-05-15T02:42:52.351ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34988, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:47074 req_id = 64848a05-87c2-49c7-9cf2-703eecf07ae8
312026-05-15T02:42:52.520ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65305, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:62272 req_id = 2f94071f-dbf7-4d8f-85ca-e5baf921efd4
322026-05-15T02:42:53.981ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:38512
332026-05-15T02:42:54.048ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 9b23e930-6152-46d4-8f99-3056da0e114a
342026-05-15T02:42:54.048ZINFOdns-server (store): updated generation new_generation = 1 req_id = 9b23e930-6152-46d4-8f99-3056da0e114a
352026-05-15T02:42:54.048ZINFOdns-server (http): request completed latency_us = 27423 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:38512 req_id = 9b23e930-6152-46d4-8f99-3056da0e114a response_code = 204 uri = /config
362026-05-15T02:42:54.154ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17504, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:54151 req_id = aa09059b-ca33-4029-b92b-6fb3964617b4
372026-05-15T02:42:54.402ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:54078
382026-05-15T02:42:54.411ZINFOdns-server (http): request completed latency_us = 9836 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:54078 req_id = 23c1942e-85e6-4e7d-bffd-5f939ac09d05 response_code = 200 uri = /config
392026-05-15T02:42:55.014ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:49219 req_id = 767efa75-c512-423f-a5de-05c5362e6014
402026-05-15T02:42:55.149ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:41193
412026-05-15T02:42:55.149ZINFOdns-server (http): request completed latency_us = 196 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:41193 req_id = a3bcaaec-5b23-49c1-80d3-9997aaa6b9f2 response_code = 200 uri = /config
422026-05-15T02:43:00.637ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55027, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56137 req_id = 62cceac5-b40e-41b2-9551-264a83e51c23
432026-05-15T02:43:02.715ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42736, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:14806 req_id = 5d372ea1-2673-419f-88aa-f5d7bf814ae1
442026-05-15T02:43:02.839ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5329, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:57009 req_id = 351f178e-62df-44fc-b77f-f13cab793ca4
452026-05-15T02:43:02.937ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:65241
462026-05-15T02:43:02.938ZINFOdns-server (http): request completed latency_us = 227 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:65241 req_id = e5eb2165-3d00-4ad4-8386-c19dad6e0f40 response_code = 200 uri = /config
472026-05-15T02:43:02.992ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:46137
482026-05-15T02:43:02.992ZINFOdns-server (http): request completed latency_us = 177 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:46137 req_id = 578a811a-a7ae-448d-a9e9-4d36385d7863 response_code = 200 uri = /config
492026-05-15T02:43:10.638ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15000, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63467 req_id = 21d047f3-1f81-4ec7-90ff-e99bdfef1499
502026-05-15T02:43:20.640ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18732, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61100 req_id = 34f8f904-0ba8-4e19-ac09-bd06f4f631f1
512026-05-15T02:43:20.678ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14991, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:36581 req_id = 0fa89d35-9e2c-4f83-af36-940a53eb1089
522026-05-15T02:43:20.683ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49974, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:47465 req_id = 03aa150a-282b-4b3b-a088-f23db737beb0
532026-05-15T02:43:20.688ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23676, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:65378 req_id = b2f7d231-b538-4c3d-b1c4-3f455a9ec124
542026-05-15T02:43:30.665ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35720, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63726 req_id = 9a989c5f-78c8-47cf-9d92-953ecfe3b036
552026-05-15T02:43:40.667ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4500, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60621 req_id = 313832f7-b72f-4e59-95d2-e36c9ed8543c
562026-05-15T02:43:44.154ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14962, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:34708 req_id = f6afdb90-0edd-46a2-8eea-1934138ae6c8
572026-05-15T02:43:44.158ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22826, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:52198 req_id = deac93b7-5ff0-47b6-9c94-98183011e5f4
582026-05-15T02:43:44.162ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37788, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:54248 req_id = 102e0e08-b62f-491e-a7e6-a287e6ec9f45
592026-05-15T02:43:50.668ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24061, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50844 req_id = 8c0346dc-1019-45f0-b50f-9d8e9a689756
602026-05-15T02:43:52.168ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:41588
612026-05-15T02:43:52.169ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 1dc38be8-a6d1-430c-a341-0795c52827c4
622026-05-15T02:43:52.169ZINFOdns-server (store): updated generation new_generation = 1 req_id = 1dc38be8-a6d1-430c-a341-0795c52827c4
632026-05-15T02:43:52.170ZINFOdns-server (http): request completed latency_us = 205 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:41588 req_id = 1dc38be8-a6d1-430c-a341-0795c52827c4 response_code = 204 uri = /config
642026-05-15T02:43:52.205ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:55880
652026-05-15T02:43:52.207ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 95971f66-2621-4a95-96d8-917c4fa5e5b9
662026-05-15T02:43:52.207ZINFOdns-server (store): updated generation new_generation = 1 req_id = 95971f66-2621-4a95-96d8-917c4fa5e5b9
672026-05-15T02:43:52.207ZINFOdns-server (http): request completed latency_us = 296 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:55880 req_id = 95971f66-2621-4a95-96d8-917c4fa5e5b9 response_code = 204 uri = /config
682026-05-15T02:43:52.261ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:64489
692026-05-15T02:43:52.263ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 0f3486e2-8c53-4b98-97b3-034f89866ed1
702026-05-15T02:43:52.263ZINFOdns-server (store): updated generation new_generation = 1 req_id = 0f3486e2-8c53-4b98-97b3-034f89866ed1
712026-05-15T02:43:52.263ZINFOdns-server (http): request completed latency_us = 378 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:64489 req_id = 0f3486e2-8c53-4b98-97b3-034f89866ed1 response_code = 204 uri = /config
722026-05-15T02:43:52.973ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25360, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:4113 req_id = defd9646-2cf5-462d-a99f-f0e3b3fab5b7
732026-05-15T02:43:53.068ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9324, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:64441 req_id = 26ec9625-c966-42d1-902c-a1dae9330d35
742026-05-15T02:43:53.076ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:41096
752026-05-15T02:43:53.076ZINFOdns-server (http): request completed latency_us = 267 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:41096 req_id = d3971374-5d69-4206-b8f8-751c84307d51 response_code = 200 uri = /config
762026-05-15T02:43:53.272ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:43362
772026-05-15T02:43:53.272ZINFOdns-server (http): request completed latency_us = 212 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:43362 req_id = bab31f2c-8816-4638-8e4c-368b82c934b2 response_code = 200 uri = /config
782026-05-15T02:43:53.607ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43696, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:1461 req_id = 9d35e19e-4b6f-4057-be5f-268ca494afb6
792026-05-15T02:43:53.712ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:41340
802026-05-15T02:43:53.712ZINFOdns-server (http): request completed latency_us = 188 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:41340 req_id = c022e7a5-3d18-4a70-a410-e2bbf4e7f532 response_code = 200 uri = /config
812026-05-15T02:44:00.669ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47145, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49345 req_id = 287a1e43-0be1-4f31-98fc-10f35d4b5f57
822026-05-15T02:44:10.670ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6718, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62842 req_id = 57468478-299d-4418-9d92-db1c52cdef2f
832026-05-15T02:44:20.673ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62963, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57925 req_id = 009915f2-e060-48b9-86cc-095c84fd4d9a
842026-05-15T02:44:30.674ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1069, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54888 req_id = fac0319f-f68a-48f4-9de8-921eb988391b
852026-05-15T02:44:40.677ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16577, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49714 req_id = 87cc281d-f09a-4ea6-a53f-bc3c5eecc3d8
862026-05-15T02:44:50.678ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19387, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58820 req_id = 407adbee-343f-4a67-925c-aa29de78b480
872026-05-15T02:44:51.953ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:57966
882026-05-15T02:44:51.955ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = f5c2ad43-25b1-498b-a91c-1b89a93109ba
892026-05-15T02:44:51.955ZINFOdns-server (store): updated generation new_generation = 1 req_id = f5c2ad43-25b1-498b-a91c-1b89a93109ba
902026-05-15T02:44:51.955ZINFOdns-server (http): request completed latency_us = 196 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:57966 req_id = f5c2ad43-25b1-498b-a91c-1b89a93109ba response_code = 204 uri = /config
912026-05-15T02:44:52.054ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:58598
922026-05-15T02:44:52.056ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = a0b19833-3112-4d7f-8d69-0697dcca286a
932026-05-15T02:44:52.056ZINFOdns-server (store): updated generation new_generation = 1 req_id = a0b19833-3112-4d7f-8d69-0697dcca286a
942026-05-15T02:44:52.056ZINFOdns-server (http): request completed latency_us = 237 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:58598 req_id = a0b19833-3112-4d7f-8d69-0697dcca286a response_code = 204 uri = /config
952026-05-15T02:44:52.262ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:36469
962026-05-15T02:44:52.264ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 799dbc7c-422f-4b2c-ae2f-1571288fd62b
972026-05-15T02:44:52.264ZINFOdns-server (store): updated generation new_generation = 1 req_id = 799dbc7c-422f-4b2c-ae2f-1571288fd62b
982026-05-15T02:44:52.264ZINFOdns-server (http): request completed latency_us = 285 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:36469 req_id = 799dbc7c-422f-4b2c-ae2f-1571288fd62b response_code = 204 uri = /config
992026-05-15T02:44:53.050ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27561, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:25952 req_id = 373869f2-27ae-4168-84c2-751bcf2a02bb
1002026-05-15T02:44:53.074ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12677, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:3201 req_id = 375b4a74-a41f-42f2-bce6-5da60134c6c3
1012026-05-15T02:44:53.077ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50506, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:42960 req_id = 7bf2ab0b-7518-40c6-9566-86956829ab3b
1022026-05-15T02:44:53.165ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:37689
1032026-05-15T02:44:53.165ZINFOdns-server (http): request completed latency_us = 186 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:37689 req_id = a6554fb1-9194-4f4f-86a9-9ae924f6e696 response_code = 200 uri = /config
1042026-05-15T02:44:53.203ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:39723
1052026-05-15T02:44:53.203ZINFOdns-server (http): request completed latency_us = 192 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:39723 req_id = 2dd70a29-f2b0-4399-bf28-3d609da054cc response_code = 200 uri = /config
1062026-05-15T02:44:53.243ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:36269
1072026-05-15T02:44:53.243ZINFOdns-server (http): request completed latency_us = 244 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:36269 req_id = 90aeeb30-3ee2-4c6c-b7a8-723de893f03b response_code = 200 uri = /config
1082026-05-15T02:45:00.080ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 919, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_91005d09-cc59-416f-aee6-aa65462b0fb0."), ), original: Query { name: Name("oxz_cockroachdb_91005d09-cc59-416f-aee6-aa65462b0fb0."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 49, 48, 48, 53, 100, 48, 57, 45, 99, 99, 53, 57, 45, 52, 49, 54, 102, 45, 97, 101, 101, 54, 45, 97, 97, 54, 53, 52, 54, 50, 98, 48, 102, 98, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_91005d09-cc59-416f-aee6-aa65462b0fb0." peer_addr = [fd00:1122:3344:101::7]:37181 req_id = 7c1fd5a9-7acb-4376-897a-3b5003f10ea8
1092026-05-15T02:45:00.081ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 919, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_91005d09-cc59-416f-aee6-aa65462b0fb0."), ), original: Query { name: Name("oxz_cockroachdb_91005d09-cc59-416f-aee6-aa65462b0fb0."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 49, 48, 48, 53, 100, 48, 57, 45, 99, 99, 53, 57, 45, 52, 49, 54, 102, 45, 97, 101, 101, 54, 45, 97, 97, 54, 53, 52, 54, 50, 98, 48, 102, 98, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_91005d09-cc59-416f-aee6-aa65462b0fb0." peer_addr = [fd00:1122:3344:101::7]:39588 req_id = e4488677-e524-4a13-8d8d-5bad998c5ede
1102026-05-15T02:45:00.312ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48635, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:59572 req_id = 98e471c7-cb00-41b1-8fae-46427fef5a5d
1112026-05-15T02:45:00.313ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48635, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:38609 req_id = 548fd62b-2c69-4347-859b-f5a06ce5dc71
1122026-05-15T02:45:00.442ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63464, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e193044e-aa73-41de-b4ce-b08357a3d432."), ), original: Query { name: Name("oxz_cockroachdb_e193044e-aa73-41de-b4ce-b08357a3d432."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 49, 57, 51, 48, 52, 52, 101, 45, 97, 97, 55, 51, 45, 52, 49, 100, 101, 45, 98, 52, 99, 101, 45, 98, 48, 56, 51, 53, 55, 97, 51, 100, 52, 51, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e193044e-aa73-41de-b4ce-b08357a3d432." peer_addr = [fd00:1122:3344:101::6]:41561 req_id = d3468b7c-66ba-4c56-ba6a-a2f6df51553c
1132026-05-15T02:45:00.443ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63464, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e193044e-aa73-41de-b4ce-b08357a3d432."), ), original: Query { name: Name("oxz_cockroachdb_e193044e-aa73-41de-b4ce-b08357a3d432."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 49, 57, 51, 48, 52, 52, 101, 45, 97, 97, 55, 51, 45, 52, 49, 100, 101, 45, 98, 52, 99, 101, 45, 98, 48, 56, 51, 53, 55, 97, 51, 100, 52, 51, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e193044e-aa73-41de-b4ce-b08357a3d432." peer_addr = [fd00:1122:3344:101::6]:46879 req_id = 756057c7-82a6-40f1-a9ed-2105f4b9c084
1142026-05-15T02:45:00.489ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53597, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_3bf717fa-c147-4091-a0bc-b678be30cccd."), ), original: Query { name: Name("oxz_ntp_3bf717fa-c147-4091-a0bc-b678be30cccd."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 51, 98, 102, 55, 49, 55, 102, 97, 45, 99, 49, 52, 55, 45, 52, 48, 57, 49, 45, 97, 48, 98, 99, 45, 98, 54, 55, 56, 98, 101, 51, 48, 99, 99, 99, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_3bf717fa-c147-4091-a0bc-b678be30cccd." peer_addr = [fd00:1122:3344:101::1b]:36898 req_id = 9b578f79-9604-45fe-8820-aeb2d7ffd3c0
1152026-05-15T02:45:00.569ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57924, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d92cf641-3430-4ac3-92b3-32d39db862be."), ), original: Query { name: Name("oxz_cockroachdb_d92cf641-3430-4ac3-92b3-32d39db862be."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 57, 50, 99, 102, 54, 52, 49, 45, 51, 52, 51, 48, 45, 52, 97, 99, 51, 45, 57, 50, 98, 51, 45, 51, 50, 100, 51, 57, 100, 98, 56, 54, 50, 98, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d92cf641-3430-4ac3-92b3-32d39db862be." peer_addr = [fd00:1122:3344:101::4]:44621 req_id = dacbce40-6a6c-4962-a521-e1bc4a2ed11f
1162026-05-15T02:45:00.570ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57924, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d92cf641-3430-4ac3-92b3-32d39db862be."), ), original: Query { name: Name("oxz_cockroachdb_d92cf641-3430-4ac3-92b3-32d39db862be."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 57, 50, 99, 102, 54, 52, 49, 45, 51, 52, 51, 48, 45, 52, 97, 99, 51, 45, 57, 50, 98, 51, 45, 51, 50, 100, 51, 57, 100, 98, 56, 54, 50, 98, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d92cf641-3430-4ac3-92b3-32d39db862be." peer_addr = [fd00:1122:3344:101::4]:53367 req_id = 940c9854-c3e0-47f2-a161-392cc1a61292
1172026-05-15T02:45:00.582ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26823, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b22828b8-2881-4adf-80d6-577e84d783a3."), ), original: Query { name: Name("oxz_cockroachdb_b22828b8-2881-4adf-80d6-577e84d783a3."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 50, 50, 56, 50, 56, 98, 56, 45, 50, 56, 56, 49, 45, 52, 97, 100, 102, 45, 56, 48, 100, 54, 45, 53, 55, 55, 101, 56, 52, 100, 55, 56, 51, 97, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b22828b8-2881-4adf-80d6-577e84d783a3." peer_addr = [fd00:1122:3344:101::3]:60990 req_id = e2433188-f142-44c1-9e14-5fbafdcc838d
1182026-05-15T02:45:00.583ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26823, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b22828b8-2881-4adf-80d6-577e84d783a3."), ), original: Query { name: Name("oxz_cockroachdb_b22828b8-2881-4adf-80d6-577e84d783a3."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 50, 50, 56, 50, 56, 98, 56, 45, 50, 56, 56, 49, 45, 52, 97, 100, 102, 45, 56, 48, 100, 54, 45, 53, 55, 55, 101, 56, 52, 100, 55, 56, 51, 97, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b22828b8-2881-4adf-80d6-577e84d783a3." peer_addr = [fd00:1122:3344:101::3]:35034 req_id = a3396504-3f98-45a3-b95a-389c66dd909a
1192026-05-15T02:45:00.588ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32830, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_65eec142-5807-438b-b65a-2b349a4d4cfa."), ), original: Query { name: Name("oxz_cockroachdb_65eec142-5807-438b-b65a-2b349a4d4cfa."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 53, 101, 101, 99, 49, 52, 50, 45, 53, 56, 48, 55, 45, 52, 51, 56, 98, 45, 98, 54, 53, 97, 45, 50, 98, 51, 52, 57, 97, 52, 100, 52, 99, 102, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_65eec142-5807-438b-b65a-2b349a4d4cfa." peer_addr = [fd00:1122:3344:101::5]:36981 req_id = f7d0082d-5e2f-48d4-9864-c814bcf79af8
1202026-05-15T02:45:00.589ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32830, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_65eec142-5807-438b-b65a-2b349a4d4cfa."), ), original: Query { name: Name("oxz_cockroachdb_65eec142-5807-438b-b65a-2b349a4d4cfa."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 53, 101, 101, 99, 49, 52, 50, 45, 53, 56, 48, 55, 45, 52, 51, 56, 98, 45, 98, 54, 53, 97, 45, 50, 98, 51, 52, 57, 97, 52, 100, 52, 99, 102, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_65eec142-5807-438b-b65a-2b349a4d4cfa." peer_addr = [fd00:1122:3344:101::5]:35607 req_id = 36f9bd72-3dc0-4517-bcb1-d05f4f0a2176
1212026-05-15T02:45:00.680ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41989, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64740 req_id = 5ff02a23-7560-49a8-a82b-9a87fa356efc
1222026-05-15T02:45:00.681ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29112, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), ), original: Query { name: Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 50, 49, 51, 50, 102, 101, 100, 45, 97, 54, 100, 97, 45, 52, 51, 101, 57, 45, 98, 99, 98, 48, 45, 49, 55, 102, 55, 97, 99, 55, 48, 48, 51, 48, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304." peer_addr = [fd00:1122:3344:101::e]:54320 req_id = 779ca0b1-a4a8-4503-a860-b7338b9498fe
1232026-05-15T02:45:00.682ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29112, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), ), original: Query { name: Name("oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 50, 49, 51, 50, 102, 101, 100, 45, 97, 54, 100, 97, 45, 52, 51, 101, 57, 45, 98, 99, 98, 48, 45, 49, 55, 102, 55, 97, 99, 55, 48, 48, 51, 48, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e2132fed-a6da-43e9-bcb0-17f7ac700304." peer_addr = [fd00:1122:3344:101::e]:37059 req_id = 06b6146e-a237-4980-8c93-143c50c3a379