Max level shown:
1 [ Apr 17 23:26:23 Enabled. ]
2 [ Apr 17 23:26:23 Rereading configuration. ]
3 [ Apr 17 23:26:24 Rereading configuration. ]
4 [ Apr 17 23:26:25 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:1::1]:5353 --dns-address [fd00:1122:3344:1::1]:53 &"). ]
5 [ Apr 17 23:26:25 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-04-17T23:26:25.557ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:1::1]:53 }
92026-04-17T23:26:25.557ZINFOdns-server (store): opening sled database path = /data/dns
102026-04-17T23:26:25.574ZINFOdns-server (store): pruning trees for generations newer than 0
112026-04-17T23:26:25.574ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-04-17T23:26:25.575ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:1::1]:53
132026-04-17T23:26:25.575ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:1::1]:5353
142026-04-17T23:26:26.310ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:1::2]:55116
152026-04-17T23:26:26.310ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 23a42d5d-fb36-4858-87a7-cb31104966ec
162026-04-17T23:26:26.310ZINFOdns-server (store): pruning trees for generations newer than 0
172026-04-17T23:26:26.328ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-04-17T23:26:26.328ZINFOdns-server (store): updated generation new_generation = 1 req_id = 23a42d5d-fb36-4858-87a7-cb31104966ec
192026-04-17T23:26:26.328ZINFOdns-server (http): request completed latency_us = 17794 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:1::2]:55116 req_id = 23a42d5d-fb36-4858-87a7-cb31104966ec response_code = 204 uri = /config
202026-04-17T23:26:38.755ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32439, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:38969 req_id = aef57503-9aed-4f36-bbd1-a7c1fb899ddd
212026-04-17T23:26:38.759ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18032, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:56734 req_id = da67243f-6ccd-4e31-9b34-773b673bb2e7
222026-04-17T23:26:38.764ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4463, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49461 req_id = a01cba22-f02a-4cab-a854-b0e65b3981f9
232026-04-17T23:28:28.019ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6201, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51915 req_id = 86d7072f-8a0a-427a-a91a-480e69bdcb25
242026-04-17T23:28:31.859ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:40390 req_id = 361e5a28-12fc-4a9d-8c28-edbe88068682
252026-04-17T23:28:31.873ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:44823 req_id = 2aa91e5c-a1dc-42f2-aaff-37b7aad937c1
262026-04-17T23:28:38.003ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17853, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64433 req_id = 0bf9a224-2321-4f0d-b5f1-15b45ff0a763
272026-04-17T23:28:38.195ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25243, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:58269 req_id = 94b8292e-4b20-4b72-869e-938856a2cf79
282026-04-17T23:28:38.197ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32106, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:32262 req_id = da1bb4ae-046b-47f5-9848-ce6ae58ba046
292026-04-17T23:28:38.200ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25175, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:24403 req_id = 995ad2c6-e954-4b81-b68c-082ef60d5787
302026-04-17T23:28:45.826ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12809, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:47112 req_id = e1e5bdd1-d07b-417f-9716-02ba11e07c33
312026-04-17T23:28:46.179ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:59078
322026-04-17T23:28:46.227ZINFOdns-server (http): request completed latency_us = 23102 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:59078 req_id = e81f1062-38ce-45d0-8108-bda9cd3ca34f response_code = 200 uri = /config
332026-04-17T23:28:48.005ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5281, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62460 req_id = 8bbb365a-1cc5-474f-9759-7a1aad138ffb
342026-04-17T23:28:48.170ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:37897
352026-04-17T23:28:48.177ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = b1706292-f532-4e54-b338-48da11d19699
362026-04-17T23:28:48.177ZINFOdns-server (store): updated generation new_generation = 1 req_id = b1706292-f532-4e54-b338-48da11d19699
372026-04-17T23:28:48.177ZINFOdns-server (http): request completed latency_us = 3083 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:37897 req_id = b1706292-f532-4e54-b338-48da11d19699 response_code = 204 uri = /config
382026-04-17T23:28:48.198ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51877, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:5614 req_id = fc776d33-1e53-435d-8d77-7ea5bd1400bc
392026-04-17T23:28:48.441ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:39248
402026-04-17T23:28:48.441ZINFOdns-server (http): request completed latency_us = 241 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:39248 req_id = dade0e4d-e1e2-47a4-90dc-a38f886565f4 response_code = 200 uri = /config
412026-04-17T23:28:49.197ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27204, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:11571 req_id = 70d6bda7-a289-444d-839c-402b35cc4e6e
422026-04-17T23:28:49.198ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7148, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:15838 req_id = fc7bcf0e-74d5-4c86-99b2-75b881750fed
432026-04-17T23:28:49.348ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:55538
442026-04-17T23:28:49.348ZINFOdns-server (http): request completed latency_us = 243 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:55538 req_id = 23259fd6-bd22-4430-95fd-6cf12d3e968a response_code = 200 uri = /config
452026-04-17T23:28:49.447ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:57630
462026-04-17T23:28:49.447ZINFOdns-server (http): request completed latency_us = 225 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:57630 req_id = 229ccca1-2b4a-4dc9-8bc6-52113f313dac response_code = 200 uri = /config
472026-04-17T23:28:50.785ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30302, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:54568 req_id = dc3d8756-f39b-449c-81b2-9ba875c281ac
482026-04-17T23:28:51.001ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:53378
492026-04-17T23:28:51.002ZINFOdns-server (http): request completed latency_us = 851 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:53378 req_id = 6706e85c-0491-49d8-bb29-eb1e42b60156 response_code = 200 uri = /config
502026-04-17T23:28:58.007ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24067, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59883 req_id = 3f6eac39-0d4e-4ebf-9fd5-d2c05834a1e9
512026-04-17T23:29:08.010ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20100, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50257 req_id = 6b3245be-8cd4-4ffe-9554-742ee5515d85
522026-04-17T23:29:18.011ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10385, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56455 req_id = 39cef613-9b44-4d1a-a0c3-9390b2754a8d
532026-04-17T23:29:28.012ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36765, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53676 req_id = 97d0554c-b66e-4b65-9943-de493b6682cd
542026-04-17T23:29:37.860ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:57343
552026-04-17T23:29:37.861ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = b7aaa6a1-d553-46c8-ad54-f415404d086b
562026-04-17T23:29:37.861ZINFOdns-server (store): updated generation new_generation = 1 req_id = b7aaa6a1-d553-46c8-ad54-f415404d086b
572026-04-17T23:29:37.861ZINFOdns-server (http): request completed latency_us = 477 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:57343 req_id = b7aaa6a1-d553-46c8-ad54-f415404d086b response_code = 204 uri = /config
582026-04-17T23:29:38.014ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52308, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51533 req_id = b6c41886-d273-4e45-baea-3b10c76bfbbf
592026-04-17T23:29:38.085ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:61359
602026-04-17T23:29:38.085ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 8106abe2-8915-44c4-acd4-b6756f036aa8
612026-04-17T23:29:38.085ZINFOdns-server (store): updated generation new_generation = 1 req_id = 8106abe2-8915-44c4-acd4-b6756f036aa8
622026-04-17T23:29:38.085ZINFOdns-server (http): request completed latency_us = 259 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:61359 req_id = 8106abe2-8915-44c4-acd4-b6756f036aa8 response_code = 204 uri = /config
632026-04-17T23:29:38.278ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:59844
642026-04-17T23:29:38.278ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 6f45e39a-8ee3-4f5b-9166-258a38c37a23
652026-04-17T23:29:38.278ZINFOdns-server (store): updated generation new_generation = 1 req_id = 6f45e39a-8ee3-4f5b-9166-258a38c37a23
662026-04-17T23:29:38.278ZINFOdns-server (http): request completed latency_us = 245 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:59844 req_id = 6f45e39a-8ee3-4f5b-9166-258a38c37a23 response_code = 204 uri = /config
672026-04-17T23:29:39.249ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43940, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:18708 req_id = cdfe03f7-c09c-415d-accd-cc05fffc1cdf
682026-04-17T23:29:39.309ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52993, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:15673 req_id = 1da5aad2-9bab-463c-8ea6-da655c06e202
692026-04-17T23:29:39.350ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:50536
702026-04-17T23:29:39.350ZINFOdns-server (http): request completed latency_us = 218 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:50536 req_id = a3f2bd9d-f5f9-40e4-94ea-69cdf354a9fb response_code = 200 uri = /config
712026-04-17T23:29:39.494ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:58874
722026-04-17T23:29:39.495ZINFOdns-server (http): request completed latency_us = 187 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:58874 req_id = 8ecdb34d-4f42-49e6-8bb8-4533a7d79db7 response_code = 200 uri = /config
732026-04-17T23:29:40.049ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29279, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:22227 req_id = 6f26f9b6-3128-4143-800f-cfbafea1bfa7
742026-04-17T23:29:40.234ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:54394
752026-04-17T23:29:40.235ZINFOdns-server (http): request completed latency_us = 188 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:54394 req_id = 1000c12c-697e-42ab-b336-b3c14e2bcb72 response_code = 200 uri = /config
762026-04-17T23:29:48.016ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19530, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53548 req_id = 98b94e0e-ab78-4850-892d-226f19c7ffb6
772026-04-17T23:29:58.017ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33458, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56455 req_id = 4b1d8cfa-269d-41bc-8c88-ab63a804a677
782026-04-17T23:30:00.206ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36603, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:62713 req_id = 3f092db4-2aa2-4d56-80cb-92f995cdbf79
792026-04-17T23:30:00.206ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36603, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:61095 req_id = 606b1b77-0b9b-459a-9f3d-0240313236df
802026-04-17T23:30:00.564ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), ), original: Query { name: Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 101, 98, 102, 99, 51, 52, 53, 45, 50, 55, 51, 49, 45, 52, 100, 100, 50, 45, 97, 98, 99, 53, 45, 51, 57, 55, 98, 98, 55, 52, 100, 53, 54, 50, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b." peer_addr = [fd00:1122:3344:101::3]:36367 req_id = b0eb2bc9-a62e-46a7-8288-c69cb3807586
812026-04-17T23:30:00.564ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), ), original: Query { name: Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 101, 98, 102, 99, 51, 52, 53, 45, 50, 55, 51, 49, 45, 52, 100, 100, 50, 45, 97, 98, 99, 53, 45, 51, 57, 55, 98, 98, 55, 52, 100, 53, 54, 50, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b." peer_addr = [fd00:1122:3344:101::3]:59583 req_id = 9227af4e-6da5-4d99-a488-396751f9c1c5
822026-04-17T23:30:00.569ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), ), original: Query { name: Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 50, 98, 53, 57, 54, 101, 56, 45, 100, 49, 99, 56, 45, 52, 57, 100, 57, 45, 57, 52, 48, 102, 45, 49, 48, 98, 52, 50, 50, 101, 53, 102, 101, 56, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80." peer_addr = [fd00:1122:3344:101::5]:45829 req_id = eb214740-77b1-4a1c-947d-521f8500ead8
832026-04-17T23:30:00.569ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), ), original: Query { name: Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 50, 98, 53, 57, 54, 101, 56, 45, 100, 49, 99, 56, 45, 52, 57, 100, 57, 45, 57, 52, 48, 102, 45, 49, 48, 98, 52, 50, 50, 101, 53, 102, 101, 56, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80." peer_addr = [fd00:1122:3344:101::5]:51462 req_id = 66565a7b-6954-4cc3-a4d1-e7349d399a6d
842026-04-17T23:30:00.596ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43303, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:57846 req_id = 650d90bc-a893-4810-b56d-b5f819ef21e2
852026-04-17T23:30:00.597ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43303, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:51911 req_id = fe50f8e4-79e8-4c23-b1fe-9d633dbd4207
862026-04-17T23:30:00.630ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22641, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), ), original: Query { name: Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 57, 55, 49, 101, 48, 51, 55, 45, 56, 57, 99, 99, 45, 52, 50, 99, 52, 45, 56, 53, 101, 102, 45, 102, 48, 97, 50, 54, 97, 56, 54, 51, 57, 53, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954." peer_addr = [fd00:1122:3344:101::6]:40443 req_id = bfcbc278-2cba-406d-93e7-7c0d8ae6601e
872026-04-17T23:30:00.630ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22641, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), ), original: Query { name: Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 57, 55, 49, 101, 48, 51, 55, 45, 56, 57, 99, 99, 45, 52, 50, 99, 52, 45, 56, 53, 101, 102, 45, 102, 48, 97, 50, 54, 97, 56, 54, 51, 57, 53, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954." peer_addr = [fd00:1122:3344:101::6]:40848 req_id = 221b9755-80ff-4018-a097-86d4e38c9020
882026-04-17T23:30:00.651ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56236, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), ), original: Query { name: Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 54, 102, 57, 50, 48, 100, 102, 45, 52, 52, 97, 98, 45, 52, 102, 48, 52, 45, 98, 48, 100, 101, 45, 50, 50, 53, 99, 101, 55, 54, 50, 52, 51, 55, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370." peer_addr = [fd00:1122:3344:101::7]:36498 req_id = e14fce57-de01-4936-8605-576c2132925f
892026-04-17T23:30:00.651ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56236, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), ), original: Query { name: Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 54, 102, 57, 50, 48, 100, 102, 45, 52, 52, 97, 98, 45, 52, 102, 48, 52, 45, 98, 48, 100, 101, 45, 50, 50, 53, 99, 101, 55, 54, 50, 52, 51, 55, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370." peer_addr = [fd00:1122:3344:101::7]:37131 req_id = e756e340-1ef5-474e-ab20-f39d7c0dfa48
902026-04-17T23:30:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44765, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a."), ), original: Query { name: Name("oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 51, 100, 102, 51, 54, 99, 51, 98, 45, 102, 53, 101, 100, 45, 52, 100, 54, 54, 45, 57, 99, 98, 97, 45, 56, 98, 98, 100, 51, 101, 55, 51, 55, 100, 56, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a." peer_addr = [fd00:1122:3344:101::1b]:61501 req_id = b3cb0988-8b9c-4a83-b513-1bd37372a83a
912026-04-17T23:30:00.686ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), ), original: Query { name: Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 50, 51, 99, 98, 100, 98, 57, 45, 54, 101, 50, 50, 45, 52, 51, 98, 97, 45, 56, 101, 50, 50, 45, 53, 99, 53, 49, 52, 97, 97, 52, 55, 48, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049." peer_addr = [fd00:1122:3344:101::4]:52453 req_id = 89ed4d24-680a-4283-8cf0-0bc21d148165
922026-04-17T23:30:00.687ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), ), original: Query { name: Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 50, 51, 99, 98, 100, 98, 57, 45, 54, 101, 50, 50, 45, 52, 51, 98, 97, 45, 56, 101, 50, 50, 45, 53, 99, 53, 49, 52, 97, 97, 52, 55, 48, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049." peer_addr = [fd00:1122:3344:101::4]:61945 req_id = 327843b4-e67b-4000-bf8b-7d1b5e62d482