Max level shown:
1 [ Jul 16 22:57:19 Enabled. ]
2 [ Jul 16 22:57:19 Rereading configuration. ]
3 [ Jul 16 22:57:20 Rereading configuration. ]
4 [ Jul 16 22:57:21 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Jul 16 22:57:21 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-16T22:57:21.602ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-07-16T22:57:21.602ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-16T22:57:21.625ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-16T22:57:21.625ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-16T22:57:21.625ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-07-16T22:57:21.625ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-07-16T22:57:21.644ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60382, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:4607 req_id = 642a0a9d-f4b1-4132-ba91-99d191bf9253
152026-07-16T22:57:21.653ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19666, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:47674 req_id = d7fa7cfa-8fe4-4cbe-b14d-9180d4d6b391
162026-07-16T22:57:21.660ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:57031
172026-07-16T22:57:21.661ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 77291ce7-b97c-41d1-a2db-75c438d43a2a
182026-07-16T22:57:21.661ZINFOdns-server (store): pruning trees for generations newer than 0
192026-07-16T22:57:21.675ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
202026-07-16T22:57:21.675ZINFOdns-server (store): updated generation new_generation = 1 req_id = 77291ce7-b97c-41d1-a2db-75c438d43a2a
212026-07-16T22:57:21.675ZINFOdns-server (http): request completed latency_us = 13934 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:57031 req_id = 77291ce7-b97c-41d1-a2db-75c438d43a2a response_code = 204 uri = /config
222026-07-16T22:57:33.824ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5421, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:38814 req_id = 0d5b09e4-1e1c-4496-8f8e-573831a4a531
232026-07-16T22:57:33.828ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 672, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:33325 req_id = 093659f6-986c-4304-9ee2-2e879a30406d
242026-07-16T22:57:33.833ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31222, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:54901 req_id = ea6315f3-0ecf-4174-9355-1b8332ad6336
252026-07-16T22:58:42.882ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8071, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59246 req_id = 1943b886-7362-4b08-bd41-a3a0fcbfb5b7
262026-07-16T22:58:46.257ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25732, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:48565 req_id = c93e7476-4028-4d0b-8bfe-12185db62c41
272026-07-16T22:58:46.262ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25732, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:40178 req_id = ca1c5d66-690c-40cc-9249-427f91103333
282026-07-16T22:58:51.220ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13298, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:50508 req_id = 0b8e1e3b-60ed-4843-b3e6-0213cb4b42ff
292026-07-16T22:58:51.562ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50035, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:36255 req_id = 2171d162-2882-4032-bc5e-f8bd01002809
302026-07-16T22:58:52.138ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59564, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:1052 req_id = d85fbec4-550a-440f-b673-d889ded4ecb9
312026-07-16T22:58:52.858ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65222, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58002 req_id = 697bc413-d45d-4ecd-8472-6286e38bdab0
322026-07-16T22:59:02.860ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28736, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55436 req_id = c4def5f6-97de-4db2-bcf7-d5a6eaad972b
332026-07-16T22:59:12.861ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22998, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49770 req_id = 712d3db2-1024-4ac4-bdcb-ed531464dc74
342026-07-16T22:59:22.862ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10535, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49407 req_id = cc0f5aa4-8d4e-4a80-84e2-dd6a9c27122e
352026-07-16T22:59:32.864ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58341, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54813 req_id = a3916b2f-43eb-45c2-8e89-450155e77a30
362026-07-16T22:59:40.771ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6773, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:39946 req_id = c0c4bdff-c32c-4834-b549-0f79905da43c
372026-07-16T22:59:40.980ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:57790
382026-07-16T22:59:41.032ZINFOdns-server (http): request completed latency_us = 33695 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:57790 req_id = eb90cc83-0804-4874-adf1-91745c9dca7a response_code = 200 uri = /config
392026-07-16T22:59:42.865ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16849, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58239 req_id = 2f1895a5-3e78-4c5d-9a96-461a973a3918
402026-07-16T22:59:43.767ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41031, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:1790 req_id = a7273f9d-5723-4f35-9fba-5654a8c83877
412026-07-16T22:59:43.903ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:63476
422026-07-16T22:59:43.909ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = d3bad595-1133-4b3b-8f0e-c1368c298102
432026-07-16T22:59:43.909ZINFOdns-server (store): updated generation new_generation = 1 req_id = d3bad595-1133-4b3b-8f0e-c1368c298102
442026-07-16T22:59:43.909ZINFOdns-server (http): request completed latency_us = 2582 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:63476 req_id = d3bad595-1133-4b3b-8f0e-c1368c298102 response_code = 204 uri = /config
452026-07-16T22:59:43.975ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:34965
462026-07-16T22:59:43.975ZINFOdns-server (http): request completed latency_us = 187 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:34965 req_id = 827e9f2a-9108-4689-8f99-bf9bb968fe70 response_code = 200 uri = /config
472026-07-16T22:59:44.851ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61370, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:14491 req_id = b402f399-5c97-4373-b0fc-60c7e5277d2a
482026-07-16T22:59:44.944ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:42793
492026-07-16T22:59:44.945ZINFOdns-server (http): request completed latency_us = 247 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:42793 req_id = bb35f33d-5fdf-44d5-9e85-c1c6c45e0b24 response_code = 200 uri = /config
502026-07-16T22:59:51.297ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:62791
512026-07-16T22:59:51.305ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 788fafdc-6892-429e-9e15-4632a6cf6277
522026-07-16T22:59:51.305ZINFOdns-server (store): updated generation new_generation = 1 req_id = 788fafdc-6892-429e-9e15-4632a6cf6277
532026-07-16T22:59:51.305ZINFOdns-server (http): request completed latency_us = 236 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:62791 req_id = 788fafdc-6892-429e-9e15-4632a6cf6277 response_code = 204 uri = /config
542026-07-16T22:59:51.371ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:45938
552026-07-16T22:59:51.375ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = d644d12f-8765-4c8e-a738-381f6d6c48e8
562026-07-16T22:59:51.375ZINFOdns-server (store): updated generation new_generation = 1 req_id = d644d12f-8765-4c8e-a738-381f6d6c48e8
572026-07-16T22:59:51.375ZINFOdns-server (http): request completed latency_us = 277 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:45938 req_id = d644d12f-8765-4c8e-a738-381f6d6c48e8 response_code = 204 uri = /config
582026-07-16T22:59:52.146ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:43312
592026-07-16T22:59:52.150ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 855a6a11-9c18-44a3-a11a-07ff8bfd698d
602026-07-16T22:59:52.150ZINFOdns-server (store): updated generation new_generation = 1 req_id = 855a6a11-9c18-44a3-a11a-07ff8bfd698d
612026-07-16T22:59:52.150ZINFOdns-server (http): request completed latency_us = 319 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:43312 req_id = 855a6a11-9c18-44a3-a11a-07ff8bfd698d response_code = 204 uri = /config
622026-07-16T22:59:52.165ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:19611 req_id = 9edd83f1-6b96-4c57-b9bb-4c94c9fc415a
632026-07-16T22:59:52.243ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25917, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:47350 req_id = 6466f594-75ff-4a27-94f2-e0bffe460981
642026-07-16T22:59:52.312ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:55720
652026-07-16T22:59:52.312ZINFOdns-server (http): request completed latency_us = 273 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:55720 req_id = f83fc5f0-3cb6-4088-98e2-a26b83139382 response_code = 200 uri = /config
662026-07-16T22:59:52.393ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:58446
672026-07-16T22:59:52.394ZINFOdns-server (http): request completed latency_us = 207 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:58446 req_id = 4adef14b-5665-4ab5-8f45-ac07beb22dd0 response_code = 200 uri = /config
682026-07-16T22:59:52.867ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47003, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61997 req_id = f3c6aeb6-95f5-4ca2-b116-24a2d4d1db4e
692026-07-16T22:59:53.636ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21733, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:52142 req_id = 1a986977-23b1-470f-b0c2-d498afbe124b
702026-07-16T22:59:53.738ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:48079
712026-07-16T22:59:53.738ZINFOdns-server (http): request completed latency_us = 211 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:48079 req_id = 3fa37e3f-675c-4038-a335-00218c195c25 response_code = 200 uri = /config
722026-07-16T23:00:00.146ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), ), original: Query { name: Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 54, 100, 49, 100, 99, 48, 98, 45, 52, 97, 48, 101, 45, 52, 50, 52, 55, 45, 98, 52, 98, 50, 45, 100, 99, 48, 54, 101, 99, 56, 100, 48, 57, 49, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916." peer_addr = [fd00:1122:3344:101::4]:46033 req_id = 243823fe-b7e9-4f4f-aeb3-5da30485c748
732026-07-16T23:00:00.147ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), ), original: Query { name: Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 54, 100, 49, 100, 99, 48, 98, 45, 52, 97, 48, 101, 45, 52, 50, 52, 55, 45, 98, 52, 98, 50, 45, 100, 99, 48, 54, 101, 99, 56, 100, 48, 57, 49, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916." peer_addr = [fd00:1122:3344:101::4]:36474 req_id = 51ae45f3-d7c9-4247-9210-0c9c67840dc6
742026-07-16T23:00:00.158ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), ), original: Query { name: Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 56, 101, 97, 50, 55, 57, 48, 45, 52, 97, 49, 102, 45, 52, 99, 102, 50, 45, 57, 54, 102, 48, 45, 101, 52, 55, 100, 57, 55, 99, 49, 51, 100, 99, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc." peer_addr = [fd00:1122:3344:101::6]:39540 req_id = 14de942b-42ca-4f09-b01d-b4af5190c45b
752026-07-16T23:00:00.159ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), ), original: Query { name: Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 56, 101, 97, 50, 55, 57, 48, 45, 52, 97, 49, 102, 45, 52, 99, 102, 50, 45, 57, 54, 102, 48, 45, 101, 52, 55, 100, 57, 55, 99, 49, 51, 100, 99, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc." peer_addr = [fd00:1122:3344:101::6]:44663 req_id = e551f55c-57bd-46fe-a25e-9eb9063701f9
762026-07-16T23:00:00.161ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54777, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), ), original: Query { name: Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 98, 98, 101, 53, 50, 55, 53, 45, 99, 97, 97, 50, 45, 52, 102, 98, 48, 45, 98, 54, 97, 55, 45, 51, 56, 100, 57, 56, 98, 53, 51, 56, 48, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a." peer_addr = [fd00:1122:3344:101::7]:57111 req_id = 0c112016-e16f-4678-8db9-26a254ce9fc5
772026-07-16T23:00:00.161ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54777, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), ), original: Query { name: Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 98, 98, 101, 53, 50, 55, 53, 45, 99, 97, 97, 50, 45, 52, 102, 98, 48, 45, 98, 54, 97, 55, 45, 51, 56, 100, 57, 56, 98, 53, 51, 56, 48, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a." peer_addr = [fd00:1122:3344:101::7]:35184 req_id = 09f6d2c9-43c8-477d-98dc-a71f0852a7fd
782026-07-16T23:00:00.193ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61673, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), ), original: Query { name: Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 51, 98, 97, 57, 49, 51, 98, 45, 97, 50, 57, 49, 45, 52, 51, 57, 98, 45, 56, 48, 56, 53, 45, 54, 49, 51, 97, 56, 56, 100, 102, 98, 49, 52, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140." peer_addr = [fd00:1122:3344:101::3]:46959 req_id = 625e09b4-d12f-4900-bb4f-7d121eda4720
792026-07-16T23:00:00.194ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61673, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), ), original: Query { name: Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 51, 98, 97, 57, 49, 51, 98, 45, 97, 50, 57, 49, 45, 52, 51, 57, 98, 45, 56, 48, 56, 53, 45, 54, 49, 51, 97, 56, 56, 100, 102, 98, 49, 52, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140." peer_addr = [fd00:1122:3344:101::3]:33002 req_id = 146e0ebf-28db-43f2-95bd-2d5ab7e8d2bd
802026-07-16T23:00:00.257ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55664, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), ), original: Query { name: Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 100, 54, 100, 101, 102, 48, 97, 45, 56, 50, 54, 101, 45, 52, 102, 102, 101, 45, 57, 97, 49, 53, 45, 100, 102, 97, 48, 53, 56, 98, 53, 49, 54, 57, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b." peer_addr = [fd00:1122:3344:101::5]:33847 req_id = a80eb2cf-420e-424a-b46b-d6d4a99ec592
812026-07-16T23:00:00.257ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55664, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), ), original: Query { name: Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 100, 54, 100, 101, 102, 48, 97, 45, 56, 50, 54, 101, 45, 52, 102, 102, 101, 45, 57, 97, 49, 53, 45, 100, 102, 97, 48, 53, 56, 98, 53, 49, 54, 57, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b." peer_addr = [fd00:1122:3344:101::5]:35398 req_id = a2992321-58ff-4714-832b-ff03b8b10a27
822026-07-16T23:00:00.656ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3105, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:45127 req_id = bfc81c20-2759-4f4c-b9d5-2afa3bc43cf5
832026-07-16T23:00:00.656ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3105, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:63609 req_id = f691dd82-cce1-4999-8d6a-f1c8f1c0a31a
842026-07-16T23:00:00.822ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30162, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_7d3994a0-53e1-4d66-81bc-e07591448165."), ), original: Query { name: Name("oxz_ntp_7d3994a0-53e1-4d66-81bc-e07591448165."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 55, 100, 51, 57, 57, 52, 97, 48, 45, 53, 51, 101, 49, 45, 52, 100, 54, 54, 45, 56, 49, 98, 99, 45, 101, 48, 55, 53, 57, 49, 52, 52, 56, 49, 54, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_7d3994a0-53e1-4d66-81bc-e07591448165." peer_addr = [fd00:1122:3344:101::1b]:62408 req_id = b003ef86-9b0e-40ca-b6b7-f8c16b417773
852026-07-16T23:00:01.004ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52847, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:56374 req_id = fe3af7f3-c055-4a52-8c5e-b71d829e9df5
862026-07-16T23:00:01.004ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52847, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:61466 req_id = 59dca7cf-35dc-4ba8-ac47-57e75542989e
872026-07-16T23:00:02.869ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47949, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59327 req_id = 0602d699-3a4a-43a9-aa09-7dbdf91303c1