Max level shown:
1 [ Jul 16 22:57:20 Enabled. ]
2 [ Jul 16 22:57:20 Rereading configuration. ]
3 [ Jul 16 22:57:20 Rereading configuration. ]
4 [ Jul 16 22:57:21 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Jul 16 22:57:21 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-16T22:57:21.676ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-07-16T22:57:21.676ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-16T22:57:21.697ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-16T22:57:21.697ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-16T22:57:21.697ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-07-16T22:57:21.697ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-07-16T22:57:21.755ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:46826
152026-07-16T22:57:21.756ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = b91ffbbf-e572-4012-808a-c49fe66b74d2
162026-07-16T22:57:21.756ZINFOdns-server (store): pruning trees for generations newer than 0
172026-07-16T22:57:21.761ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64704, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:31762 req_id = d44854e7-6201-486c-a7ce-0f7ec281f05e
182026-07-16T22:57:21.762ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51346, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("962ce1b3-7b14-4b7f-871b-0ecaa8aed3e2.host.control-plane.oxide.internal."), ), original: Query { name: Name("962ce1b3-7b14-4b7f-871b-0ecaa8aed3e2.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 57, 54, 50, 99, 101, 49, 98, 51, 45, 55, 98, 49, 52, 45, 52, 98, 55, 102, 45, 56, 55, 49, 98, 45, 48, 101, 99, 97, 97, 56, 97, 101, 100, 51, 101, 50, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "962ce1b3-7b14-4b7f-871b-0ecaa8aed3e2.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:60711 req_id = 82f343a4-ad2b-49b5-bc35-d42b5e10993e
192026-07-16T22:57:21.762ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22946, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("5a843817-2e7d-4ea5-a8bd-48c818468ac9.host.control-plane.oxide.internal."), ), original: Query { name: Name("5a843817-2e7d-4ea5-a8bd-48c818468ac9.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 53, 97, 56, 52, 51, 56, 49, 55, 45, 50, 101, 55, 100, 45, 52, 101, 97, 53, 45, 97, 56, 98, 100, 45, 52, 56, 99, 56, 49, 56, 52, 54, 56, 97, 99, 57, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "5a843817-2e7d-4ea5-a8bd-48c818468ac9.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:34711 req_id = 65b0c53b-b69c-4b90-98d0-a2d9aa9aab51
202026-07-16T22:57:21.762ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43743, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("965d07ec-1757-4f79-8aa9-58302f827817.host.control-plane.oxide.internal."), ), original: Query { name: Name("965d07ec-1757-4f79-8aa9-58302f827817.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 57, 54, 53, 100, 48, 55, 101, 99, 45, 49, 55, 53, 55, 45, 52, 102, 55, 57, 45, 56, 97, 97, 57, 45, 53, 56, 51, 48, 50, 102, 56, 50, 55, 56, 49, 55, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "965d07ec-1757-4f79-8aa9-58302f827817.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:6118 req_id = abeec750-fa3d-4a2d-846a-684771ada64a
212026-07-16T22:57:21.769ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
222026-07-16T22:57:21.769ZINFOdns-server (store): updated generation new_generation = 1 req_id = b91ffbbf-e572-4012-808a-c49fe66b74d2
232026-07-16T22:57:21.769ZINFOdns-server (http): request completed latency_us = 13515 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:46826 req_id = b91ffbbf-e572-4012-808a-c49fe66b74d2 response_code = 204 uri = /config
242026-07-16T22:57:33.824ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5421, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:61326 req_id = 9761a380-832d-4491-aa79-6518b81fc79a
252026-07-16T22:57:33.828ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 672, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:56344 req_id = ba4c4112-165a-48f6-895c-a0530f672048
262026-07-16T22:57:33.833ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31222, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:52602 req_id = 61c765d6-95a5-48e4-bdc9-a978e44c344b
272026-07-16T22:58:42.875ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9136, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55263 req_id = 6f043b98-0212-48e0-b733-0df81eafbc08
282026-07-16T22:58:46.261ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25732, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:64623 req_id = fbfcc1b6-d9f9-44cc-900d-6859e2d5eb7d
292026-07-16T22:58:46.262ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25732, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:46198 req_id = 22df0d42-c502-4b02-ba1e-6647bbe050d4
302026-07-16T22:58:51.220ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13341, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:24586 req_id = b782a17c-61f7-4f23-9ca9-fa7dbbd3c7b2
312026-07-16T22:58:51.562ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18905, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:25954 req_id = 63720b27-2145-40c1-a8ec-e05e83ae13d7
322026-07-16T22:58:52.139ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41919, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:44502 req_id = b1a58fa5-7a75-47f1-9f6f-35f504608e90
332026-07-16T22:58:52.858ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22727, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50218 req_id = 67a43f20-ec83-4c9a-87eb-40e05bc727ba
342026-07-16T22:59:02.860ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13020, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56403 req_id = f8334853-cb2f-4e6f-9fe4-f4b92a67a9da
352026-07-16T22:59:12.861ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32977, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49484 req_id = 69fdc4b4-73b0-47ef-82a7-69b655a73b66
362026-07-16T22:59:22.862ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39606, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53436 req_id = cd784363-07df-4199-99cf-62cc9795a147
372026-07-16T22:59:32.864ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50427, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49496 req_id = e26ffb17-6ffe-4a19-872e-d9234d86b9ea
382026-07-16T22:59:40.771ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16446, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:18237 req_id = 3d5bdd90-05c3-4912-a169-d05e0fac0f86
392026-07-16T22:59:41.122ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:35017
402026-07-16T22:59:41.176ZINFOdns-server (http): request completed latency_us = 35132 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:35017 req_id = cdc209d3-2b60-4e27-a46c-d0deb6cf1512 response_code = 200 uri = /config
412026-07-16T22:59:42.865ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41283, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51111 req_id = addc239e-0856-4a89-9d7c-da3721d38027
422026-07-16T22:59:43.767ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18917, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:22556 req_id = f4dc25c4-6e0f-4666-869e-cdfef02065b1
432026-07-16T22:59:43.906ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:46116
442026-07-16T22:59:43.910ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = d63af782-e785-4839-a4f8-c30b5d721827
452026-07-16T22:59:43.910ZINFOdns-server (store): updated generation new_generation = 1 req_id = d63af782-e785-4839-a4f8-c30b5d721827
462026-07-16T22:59:43.910ZINFOdns-server (http): request completed latency_us = 3374 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:46116 req_id = d63af782-e785-4839-a4f8-c30b5d721827 response_code = 204 uri = /config
472026-07-16T22:59:43.977ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:39724
482026-07-16T22:59:43.977ZINFOdns-server (http): request completed latency_us = 230 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:39724 req_id = 0cd6cb79-4be1-4afe-b9c1-5fac711e63fd response_code = 200 uri = /config
492026-07-16T22:59:44.851ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43000, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:12068 req_id = cf0e8b34-c3df-49ec-9d71-8930304f99e7
502026-07-16T22:59:44.946ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:51270
512026-07-16T22:59:44.946ZINFOdns-server (http): request completed latency_us = 259 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:51270 req_id = b85b23a9-271a-4a15-a5e5-8708b529c17c response_code = 200 uri = /config
522026-07-16T22:59:51.305ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:52364
532026-07-16T22:59:51.305ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = dbed84af-83d2-4ee8-bb9d-b894e0d53b19
542026-07-16T22:59:51.305ZINFOdns-server (store): updated generation new_generation = 1 req_id = dbed84af-83d2-4ee8-bb9d-b894e0d53b19
552026-07-16T22:59:51.305ZINFOdns-server (http): request completed latency_us = 227 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:52364 req_id = dbed84af-83d2-4ee8-bb9d-b894e0d53b19 response_code = 204 uri = /config
562026-07-16T22:59:51.374ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:43067
572026-07-16T22:59:51.375ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 78bdd8e5-de18-4020-9021-e89e7150ce9f
582026-07-16T22:59:51.375ZINFOdns-server (store): updated generation new_generation = 1 req_id = 78bdd8e5-de18-4020-9021-e89e7150ce9f
592026-07-16T22:59:51.375ZINFOdns-server (http): request completed latency_us = 313 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:43067 req_id = 78bdd8e5-de18-4020-9021-e89e7150ce9f response_code = 204 uri = /config
602026-07-16T22:59:52.149ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:36406
612026-07-16T22:59:52.150ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = fd85e58f-8a52-4af3-8079-86ceba5c113c
622026-07-16T22:59:52.150ZINFOdns-server (store): updated generation new_generation = 1 req_id = fd85e58f-8a52-4af3-8079-86ceba5c113c
632026-07-16T22:59:52.150ZINFOdns-server (http): request completed latency_us = 321 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:36406 req_id = fd85e58f-8a52-4af3-8079-86ceba5c113c response_code = 204 uri = /config
642026-07-16T22:59:52.165ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10575, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:36934 req_id = 5dcce8c7-98d2-4111-86d4-8f8413505235
652026-07-16T22:59:52.243ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31678, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:38772 req_id = a74050e8-a566-44e4-bbb0-0adf153a41e4
662026-07-16T22:59:52.314ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:36914
672026-07-16T22:59:52.314ZINFOdns-server (http): request completed latency_us = 220 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:36914 req_id = 3b17d07b-67b1-498c-821c-94b0db5bbf14 response_code = 200 uri = /config
682026-07-16T22:59:52.395ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:57876
692026-07-16T22:59:52.395ZINFOdns-server (http): request completed latency_us = 230 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:57876 req_id = 0efb8edf-2aee-4518-8f97-b445f6b93714 response_code = 200 uri = /config
702026-07-16T22:59:52.867ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38500, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51851 req_id = fe3dea75-ee05-4ebd-82e5-08245cbcaee7
712026-07-16T22:59:53.636ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24510, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:1111 req_id = 33cf17ef-8280-455c-9d41-8cab233809ac
722026-07-16T22:59:53.740ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:35834
732026-07-16T22:59:53.740ZINFOdns-server (http): request completed latency_us = 188 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:35834 req_id = ec37c900-8245-41fb-b765-8c10dd947030 response_code = 200 uri = /config
742026-07-16T23:00:00.146ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), ), original: Query { name: Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 54, 100, 49, 100, 99, 48, 98, 45, 52, 97, 48, 101, 45, 52, 50, 52, 55, 45, 98, 52, 98, 50, 45, 100, 99, 48, 54, 101, 99, 56, 100, 48, 57, 49, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916." peer_addr = [fd00:1122:3344:101::4]:36715 req_id = bba24166-5f6f-4c20-b53a-81c923342d04
752026-07-16T23:00:00.147ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), ), original: Query { name: Name("oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 54, 100, 49, 100, 99, 48, 98, 45, 52, 97, 48, 101, 45, 52, 50, 52, 55, 45, 98, 52, 98, 50, 45, 100, 99, 48, 54, 101, 99, 56, 100, 48, 57, 49, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_36d1dc0b-4a0e-4247-b4b2-dc06ec8d0916." peer_addr = [fd00:1122:3344:101::4]:53865 req_id = a9b44d14-089f-45e1-a007-32f5d9601ad4
762026-07-16T23:00:00.158ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), ), original: Query { name: Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 56, 101, 97, 50, 55, 57, 48, 45, 52, 97, 49, 102, 45, 52, 99, 102, 50, 45, 57, 54, 102, 48, 45, 101, 52, 55, 100, 57, 55, 99, 49, 51, 100, 99, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc." peer_addr = [fd00:1122:3344:101::6]:56402 req_id = 12c6f32c-0a8c-42ff-a193-492f2e12635c
772026-07-16T23:00:00.159ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), ), original: Query { name: Name("oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 56, 101, 97, 50, 55, 57, 48, 45, 52, 97, 49, 102, 45, 52, 99, 102, 50, 45, 57, 54, 102, 48, 45, 101, 52, 55, 100, 57, 55, 99, 49, 51, 100, 99, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_58ea2790-4a1f-4cf2-96f0-e47d97c13dcc." peer_addr = [fd00:1122:3344:101::6]:33093 req_id = d58ac0b5-6bc4-41c3-9bee-eb32e8fac509
782026-07-16T23:00:00.161ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54777, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), ), original: Query { name: Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 98, 98, 101, 53, 50, 55, 53, 45, 99, 97, 97, 50, 45, 52, 102, 98, 48, 45, 98, 54, 97, 55, 45, 51, 56, 100, 57, 56, 98, 53, 51, 56, 48, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a." peer_addr = [fd00:1122:3344:101::7]:53450 req_id = 865abc52-eab2-48ef-9289-b509458be0f3
792026-07-16T23:00:00.161ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54777, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), ), original: Query { name: Name("oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 98, 98, 101, 53, 50, 55, 53, 45, 99, 97, 97, 50, 45, 52, 102, 98, 48, 45, 98, 54, 97, 55, 45, 51, 56, 100, 57, 56, 98, 53, 51, 56, 48, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5bbe5275-caa2-4fb0-b6a7-38d98b53802a." peer_addr = [fd00:1122:3344:101::7]:56406 req_id = be08d1a2-838b-4ceb-a67c-d935b8c7c141
802026-07-16T23:00:00.193ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61673, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), ), original: Query { name: Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 51, 98, 97, 57, 49, 51, 98, 45, 97, 50, 57, 49, 45, 52, 51, 57, 98, 45, 56, 48, 56, 53, 45, 54, 49, 51, 97, 56, 56, 100, 102, 98, 49, 52, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140." peer_addr = [fd00:1122:3344:101::3]:40499 req_id = 9e770e5e-0610-460d-97d5-94d6e6f04b9a
812026-07-16T23:00:00.194ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61673, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), ), original: Query { name: Name("oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 51, 98, 97, 57, 49, 51, 98, 45, 97, 50, 57, 49, 45, 52, 51, 57, 98, 45, 56, 48, 56, 53, 45, 54, 49, 51, 97, 56, 56, 100, 102, 98, 49, 52, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_93ba913b-a291-439b-8085-613a88dfb140." peer_addr = [fd00:1122:3344:101::3]:61424 req_id = f6648623-a822-4de5-a3d8-e18ae6e309e7
822026-07-16T23:00:00.257ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55664, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), ), original: Query { name: Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 100, 54, 100, 101, 102, 48, 97, 45, 56, 50, 54, 101, 45, 52, 102, 102, 101, 45, 57, 97, 49, 53, 45, 100, 102, 97, 48, 53, 56, 98, 53, 49, 54, 57, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b." peer_addr = [fd00:1122:3344:101::5]:36630 req_id = 87858715-3729-4d5e-808d-839c4bac0c11
832026-07-16T23:00:00.257ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55664, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), ), original: Query { name: Name("oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 57, 100, 54, 100, 101, 102, 48, 97, 45, 56, 50, 54, 101, 45, 52, 102, 102, 101, 45, 57, 97, 49, 53, 45, 100, 102, 97, 48, 53, 56, 98, 53, 49, 54, 57, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_9d6def0a-826e-4ffe-9a15-dfa058b5169b." peer_addr = [fd00:1122:3344:101::5]:58310 req_id = 3f81bcf7-6a00-4a89-9763-d3738d1ce6e7
842026-07-16T23:00:00.656ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3105, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:42212 req_id = 756354f9-5c1b-4b09-9432-74e08dd6eddd
852026-07-16T23:00:00.657ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3105, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:63052 req_id = f4d6beb3-e259-4b7d-af46-da08c18311d6
862026-07-16T23:00:00.822ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30162, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_7d3994a0-53e1-4d66-81bc-e07591448165."), ), original: Query { name: Name("oxz_ntp_7d3994a0-53e1-4d66-81bc-e07591448165."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 55, 100, 51, 57, 57, 52, 97, 48, 45, 53, 51, 101, 49, 45, 52, 100, 54, 54, 45, 56, 49, 98, 99, 45, 101, 48, 55, 53, 57, 49, 52, 52, 56, 49, 54, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_7d3994a0-53e1-4d66-81bc-e07591448165." peer_addr = [fd00:1122:3344:101::1b]:33571 req_id = aa0facca-1e6b-47f6-851c-f1d25b14aac6
872026-07-16T23:00:01.004ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52847, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:52073 req_id = 89d59a04-bfa4-43bb-bb45-a5eb2df02487
882026-07-16T23:00:01.004ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52847, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), ), original: Query { name: Name("oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 98, 101, 101, 99, 97, 99, 51, 48, 45, 54, 98, 53, 102, 45, 52, 98, 97, 52, 45, 56, 48, 50, 56, 45, 48, 53, 97, 53, 100, 55, 100, 50, 57, 53, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_beecac30-6b5f-4ba4-8028-05a5d7d29522." peer_addr = [fd00:1122:3344:101::e]:39595 req_id = 4084943e-9f06-466e-a357-76f032062505