Max level shown:
1 [ Jul 29 16:39:06 Enabled. ]
2 [ Jul 29 16:39:06 Rereading configuration. ]
3 [ Jul 29 16:39:07 Rereading configuration. ]
4 [ Jul 29 16:39:08 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Jul 29 16:39:08 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-29T16:39:08.220ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-07-29T16:39:08.220ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-29T16:39:08.240ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-29T16:39:08.240ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-29T16:39:08.240ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-07-29T16:39:08.241ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-07-29T16:39:08.388ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:55911
152026-07-29T16:39:08.389ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 48b87ee0-2814-4927-9259-3bafe3297ef5
162026-07-29T16:39:08.389ZINFOdns-server (store): pruning trees for generations newer than 0
172026-07-29T16:39:08.401ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-07-29T16:39:08.401ZINFOdns-server (store): updated generation new_generation = 1 req_id = 48b87ee0-2814-4927-9259-3bafe3297ef5
192026-07-29T16:39:08.401ZINFOdns-server (http): request completed latency_us = 12712 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:55911 req_id = 48b87ee0-2814-4927-9259-3bafe3297ef5 response_code = 204 uri = /config
202026-07-29T16:39:20.696ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57074, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:62684 req_id = 89ffb3c6-b861-406d-917b-ffee3b226728
212026-07-29T16:39:20.700ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50324, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:42580 req_id = 6c15e493-6e9a-49bb-8bc9-ea3c48335a55
222026-07-29T16:39:20.704ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56411, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:59541 req_id = 8c8f20cc-491c-4e8e-95fb-f805c981d556
232026-07-29T16:41:20.693ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39323, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:55489 req_id = abdedd02-1970-4cc7-a599-0968dc113179
242026-07-29T16:41:20.698ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63584, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:36479 req_id = 254419f6-f62f-497c-ab33-4da1802742b0
252026-07-29T16:41:20.703ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:58934 req_id = 07160090-d0ae-45e4-b3b4-228d27bf27d1
262026-07-29T16:42:41.375ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11878, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57464 req_id = e97c49cc-81f7-4c57-af06-7a64e0bb9816
272026-07-29T16:42:51.357ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51442, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64804 req_id = c79380ee-5742-4a30-9b3b-3c9706e13c15
282026-07-29T16:42:54.390ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48370, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:3636 req_id = 7834faae-af6c-46c7-befb-e6adf173afdd
292026-07-29T16:42:55.233ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7076, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:24149 req_id = 5e69ed01-0af8-426d-ab39-df5f3b5991e4
302026-07-29T16:42:55.472ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35395, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:18623 req_id = 809befed-5de1-490e-9dcb-a06c3168250f
312026-07-29T16:42:56.749ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:42523
322026-07-29T16:42:56.760ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:36508
332026-07-29T16:42:56.812ZINFOdns-server (http): request completed latency_us = 29601 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:36508 req_id = 6e5e742b-d230-4e88-8e4f-7410ec73ec2a response_code = 200 uri = /config
342026-07-29T16:42:56.812ZINFOdns-server (http): request completed latency_us = 29599 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:42523 req_id = a1340330-67bd-4b33-b24d-cc735d069278 response_code = 200 uri = /config
352026-07-29T16:42:57.319ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30949, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:33823 req_id = 4ad45f45-8709-4bb6-9f64-181ccbe79875
362026-07-29T16:42:57.325ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30949, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:45009 req_id = 2a4ce5a2-a64e-4947-8dfb-6d5ea53229eb
372026-07-29T16:42:57.998ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12445, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:7836 req_id = 628e5ccd-86e5-443e-8cc8-bf845b95c05f
382026-07-29T16:42:58.226ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:34064
392026-07-29T16:42:58.228ZINFOdns-server (http): request completed latency_us = 198 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:34064 req_id = 19e566a6-8bd5-4fdc-88bc-8b970bd25596 response_code = 200 uri = /config
402026-07-29T16:42:59.159ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35960, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:23919 req_id = b23874d0-3172-43bb-a45c-1a13f2b1ef40
412026-07-29T16:42:59.243ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:52973
422026-07-29T16:42:59.244ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = b9e8c503-412b-433d-9b4e-dccd65abdd10
432026-07-29T16:42:59.244ZINFOdns-server (store): updated generation new_generation = 1 req_id = b9e8c503-412b-433d-9b4e-dccd65abdd10
442026-07-29T16:42:59.244ZINFOdns-server (http): request completed latency_us = 427 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:52973 req_id = b9e8c503-412b-433d-9b4e-dccd65abdd10 response_code = 204 uri = /config
452026-07-29T16:42:59.299ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:46167
462026-07-29T16:42:59.300ZINFOdns-server (http): request completed latency_us = 265 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:46167 req_id = b5dda8ba-ded5-46d3-8a2e-d6be097027d2 response_code = 200 uri = /config
472026-07-29T16:43:00.468ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53684, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:5395 req_id = 40f3526d-b311-421d-845d-f519a6863a77
482026-07-29T16:43:00.595ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:42154
492026-07-29T16:43:00.595ZINFOdns-server (http): request completed latency_us = 218 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:42154 req_id = 089e6afc-cb3a-45c1-8221-2dd8ca5fd9cf response_code = 200 uri = /config
502026-07-29T16:43:01.358ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8531, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49824 req_id = 6e7774dc-2d74-4aff-82da-bff641281b32
512026-07-29T16:43:05.523ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54780, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:14488 req_id = 18886619-0924-49c4-9d13-571cb9875985
522026-07-29T16:43:05.825ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:49486
532026-07-29T16:43:05.826ZINFOdns-server (http): request completed latency_us = 204 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:49486 req_id = d2e0091e-e24e-4daa-9341-d17d20bf6cf4 response_code = 200 uri = /config
542026-07-29T16:43:06.404ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20130, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:1548 req_id = 2ad888a9-dd9a-402c-a5a2-729cf81cb49a
552026-07-29T16:43:06.516ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:48656
562026-07-29T16:43:06.516ZINFOdns-server (http): request completed latency_us = 188 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:48656 req_id = 66210cc4-a396-40be-a31f-4d7a6726349a response_code = 200 uri = /config
572026-07-29T16:43:11.361ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7400, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60178 req_id = 73b996e5-156d-4f6d-946b-e0a141722b11
582026-07-29T16:43:21.362ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54071, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64684 req_id = 0daebca4-4b9d-47dc-a0a0-56ff436d1521
592026-07-29T16:43:31.363ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4152, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64640 req_id = 4335d400-ef11-41d0-b09f-977400d10756
602026-07-29T16:43:41.366ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42339, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52812 req_id = fa620f50-fa68-48b6-96a0-e2b18d92b605
612026-07-29T16:43:51.367ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33978, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52823 req_id = 8921c875-98ca-4fbc-8332-2308abf81e99
622026-07-29T16:43:54.210ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:53378
632026-07-29T16:43:54.210ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = e757d896-1b16-48ff-b98a-588c02298cf9
642026-07-29T16:43:54.210ZINFOdns-server (store): updated generation new_generation = 1 req_id = e757d896-1b16-48ff-b98a-588c02298cf9
652026-07-29T16:43:54.210ZINFOdns-server (http): request completed latency_us = 357 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:53378 req_id = e757d896-1b16-48ff-b98a-588c02298cf9 response_code = 204 uri = /config
662026-07-29T16:43:55.103ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:62523
672026-07-29T16:43:55.104ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 2914b314-6dcf-460f-ac70-a9af3523ebbd
682026-07-29T16:43:55.104ZINFOdns-server (store): updated generation new_generation = 1 req_id = 2914b314-6dcf-460f-ac70-a9af3523ebbd
692026-07-29T16:43:55.104ZINFOdns-server (http): request completed latency_us = 199 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:62523 req_id = 2914b314-6dcf-460f-ac70-a9af3523ebbd response_code = 204 uri = /config
702026-07-29T16:43:55.438ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:37019
712026-07-29T16:43:55.439ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 7623dc55-352c-4559-aeed-073211983c1e
722026-07-29T16:43:55.439ZINFOdns-server (store): updated generation new_generation = 1 req_id = 7623dc55-352c-4559-aeed-073211983c1e
732026-07-29T16:43:55.439ZINFOdns-server (http): request completed latency_us = 286 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:37019 req_id = 7623dc55-352c-4559-aeed-073211983c1e response_code = 204 uri = /config
742026-07-29T16:43:55.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40174, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:19768 req_id = e714c0ee-01a5-4a65-ade2-1d884e73c6b3
752026-07-29T16:43:55.746ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:41609
762026-07-29T16:43:55.746ZINFOdns-server (http): request completed latency_us = 196 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:41609 req_id = 0bdab375-b4f3-4cfd-9273-4b401449cfa2 response_code = 200 uri = /config
772026-07-29T16:43:56.154ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24372, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:33544 req_id = b767422d-a087-4f45-a2a7-0081f32c5e9f
782026-07-29T16:43:56.258ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:53202
792026-07-29T16:43:56.259ZINFOdns-server (http): request completed latency_us = 247 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:53202 req_id = 89de7500-8303-4c7e-a664-899552b52cd4 response_code = 200 uri = /config
802026-07-29T16:43:56.579ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24724, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:63932 req_id = aa3d8792-fff2-40e2-83d7-a6eaeb1a4877
812026-07-29T16:43:56.773ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:44710
822026-07-29T16:43:56.773ZINFOdns-server (http): request completed latency_us = 215 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:44710 req_id = f77df915-08c0-4ada-914b-e36899771097 response_code = 200 uri = /config
832026-07-29T16:44:01.369ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51666, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54550 req_id = aa949d8a-7604-4f00-8295-db74b93bb02f
842026-07-29T16:44:11.370ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8363, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58070 req_id = 762eb082-0d9a-4475-b4ad-b566de422372
852026-07-29T16:44:21.372ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54367, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60503 req_id = 2c1d72c6-43e1-4779-8957-37fc71619502
862026-07-29T16:44:31.375ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55113, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64706 req_id = 674adde4-bc73-42cf-8762-ea61e42fbadc
872026-07-29T16:44:41.376ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18722, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63603 req_id = d5d221d6-4ac0-4e70-9003-82675f67f5b5
882026-07-29T16:44:51.378ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17191, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51231 req_id = d529930c-e322-4528-9443-b9ca4a7cb5fd
892026-07-29T16:44:54.125ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:62759
902026-07-29T16:44:54.125ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 2718cdfb-dc6e-432e-be18-54c76cf81b1c
912026-07-29T16:44:54.125ZINFOdns-server (store): updated generation new_generation = 1 req_id = 2718cdfb-dc6e-432e-be18-54c76cf81b1c
922026-07-29T16:44:54.125ZINFOdns-server (http): request completed latency_us = 183 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:62759 req_id = 2718cdfb-dc6e-432e-be18-54c76cf81b1c response_code = 204 uri = /config
932026-07-29T16:44:54.895ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65411, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:40438 req_id = 22c86c4b-b605-4a08-a78f-290ee1efd324
942026-07-29T16:44:54.995ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:36991
952026-07-29T16:44:54.995ZINFOdns-server (http): request completed latency_us = 204 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:36991 req_id = 2fa1e3fc-4f7a-4d93-9a14-4b972fc04edb response_code = 200 uri = /config
962026-07-29T16:44:55.102ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:46047
972026-07-29T16:44:55.103ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 6e1f2310-df8b-4cb0-b478-71b99e637ed0
982026-07-29T16:44:55.103ZINFOdns-server (store): updated generation new_generation = 1 req_id = 6e1f2310-df8b-4cb0-b478-71b99e637ed0
992026-07-29T16:44:55.103ZINFOdns-server (http): request completed latency_us = 236 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:46047 req_id = 6e1f2310-df8b-4cb0-b478-71b99e637ed0 response_code = 204 uri = /config
1002026-07-29T16:44:55.201ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:40345
1012026-07-29T16:44:55.201ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 0406388d-b989-467d-8884-efa583ea10ad
1022026-07-29T16:44:55.201ZINFOdns-server (store): updated generation new_generation = 1 req_id = 0406388d-b989-467d-8884-efa583ea10ad
1032026-07-29T16:44:55.201ZINFOdns-server (http): request completed latency_us = 250 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:40345 req_id = 0406388d-b989-467d-8884-efa583ea10ad response_code = 204 uri = /config
1042026-07-29T16:44:56.395ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19058, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:26092 req_id = 086de71e-f638-42ec-af2a-f67484517840
1052026-07-29T16:44:56.400ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23422, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:40762 req_id = 21f437a1-501e-40ec-9823-372cb12ce934
1062026-07-29T16:44:56.502ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:51200
1072026-07-29T16:44:56.503ZINFOdns-server (http): request completed latency_us = 214 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:51200 req_id = 506a71e5-3d00-48bf-82df-7f5f3fba79f3 response_code = 200 uri = /config
1082026-07-29T16:44:56.593ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:34751
1092026-07-29T16:44:56.593ZINFOdns-server (http): request completed latency_us = 258 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:34751 req_id = af022b5a-cd67-4980-bdd8-c28f9c781c9b response_code = 200 uri = /config
1102026-07-29T16:45:00.034ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), ), original: Query { name: Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 49, 57, 57, 51, 55, 54, 102, 45, 57, 102, 98, 101, 45, 52, 101, 56, 100, 45, 97, 99, 48, 102, 45, 50, 99, 49, 50, 49, 100, 48, 51, 54, 55, 54, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e." peer_addr = [fd00:1122:3344:101::3]:41329 req_id = 60673637-5c49-44a9-9114-1729d42883b6
1112026-07-29T16:45:00.035ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), ), original: Query { name: Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 49, 57, 57, 51, 55, 54, 102, 45, 57, 102, 98, 101, 45, 52, 101, 56, 100, 45, 97, 99, 48, 102, 45, 50, 99, 49, 50, 49, 100, 48, 51, 54, 55, 54, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e." peer_addr = [fd00:1122:3344:101::3]:45901 req_id = ff737ff3-afc4-4861-a4c0-7c228ce5d42c
1122026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54786, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), ), original: Query { name: Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 56, 56, 50, 102, 97, 102, 98, 45, 54, 98, 51, 49, 45, 52, 101, 98, 55, 45, 97, 99, 54, 50, 45, 52, 50, 53, 53, 97, 101, 100, 51, 99, 48, 56, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c." peer_addr = [fd00:1122:3344:101::5]:59997 req_id = 94e26d05-6905-43e2-9f38-737da6eebc3d
1132026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), ), original: Query { name: Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 56, 57, 97, 51, 53, 98, 57, 45, 101, 98, 54, 101, 45, 52, 48, 56, 97, 45, 97, 48, 49, 97, 45, 101, 57, 56, 51, 57, 97, 98, 51, 52, 99, 97, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8." peer_addr = [fd00:1122:3344:101::4]:49198 req_id = 546c4c4a-1748-4c88-8be8-8fc302080bd1
1142026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54786, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), ), original: Query { name: Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 56, 56, 50, 102, 97, 102, 98, 45, 54, 98, 51, 49, 45, 52, 101, 98, 55, 45, 97, 99, 54, 50, 45, 52, 50, 53, 53, 97, 101, 100, 51, 99, 48, 56, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c." peer_addr = [fd00:1122:3344:101::5]:42201 req_id = b92badaa-ad48-44dc-a1e2-c11bf6ea131d
1152026-07-29T16:45:00.088ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), ), original: Query { name: Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 56, 57, 97, 51, 53, 98, 57, 45, 101, 98, 54, 101, 45, 52, 48, 56, 97, 45, 97, 48, 49, 97, 45, 101, 57, 56, 51, 57, 97, 98, 51, 52, 99, 97, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8." peer_addr = [fd00:1122:3344:101::4]:43629 req_id = e0a1cea1-b765-4878-985d-935eef930ceb
1162026-07-29T16:45:00.089ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), ), original: Query { name: Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 53, 98, 54, 52, 54, 48, 53, 45, 51, 57, 48, 49, 45, 52, 48, 100, 99, 45, 98, 48, 57, 48, 45, 56, 99, 49, 48, 56, 54, 56, 53, 57, 49, 102, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff." peer_addr = [fd00:1122:3344:101::7]:64294 req_id = 3d74e2b4-f892-4282-9b1b-daad277b66f0
1172026-07-29T16:45:00.090ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), ), original: Query { name: Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 53, 98, 54, 52, 54, 48, 53, 45, 51, 57, 48, 49, 45, 52, 48, 100, 99, 45, 98, 48, 57, 48, 45, 56, 99, 49, 48, 56, 54, 56, 53, 57, 49, 102, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff." peer_addr = [fd00:1122:3344:101::7]:53059 req_id = f24da694-14c2-45a8-9ca4-c7ac12d41820
1182026-07-29T16:45:00.633ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:39791 req_id = c187f154-b23d-4f7b-8b14-90521971d6dc
1192026-07-29T16:45:00.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:59268 req_id = 51fba169-6f8e-434c-9791-7210dbe41d64
1202026-07-29T16:45:01.013ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), ), original: Query { name: Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 102, 57, 53, 55, 101, 98, 101, 45, 53, 101, 52, 57, 45, 52, 57, 55, 55, 45, 97, 52, 50, 101, 45, 49, 53, 51, 52, 54, 52, 98, 51, 99, 56, 52, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847." peer_addr = [fd00:1122:3344:101::6]:47813 req_id = 389eb65a-460a-4254-ae88-6e6c405f635e
1212026-07-29T16:45:01.013ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), ), original: Query { name: Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 102, 57, 53, 55, 101, 98, 101, 45, 53, 101, 52, 57, 45, 52, 57, 55, 55, 45, 97, 52, 50, 101, 45, 49, 53, 51, 52, 54, 52, 98, 51, 99, 56, 52, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847." peer_addr = [fd00:1122:3344:101::6]:34180 req_id = 7ec778e6-bf6b-42a4-acb9-accc093c1292
1222026-07-29T16:45:01.110ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11174, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c."), ), original: Query { name: Name("oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 52, 53, 97, 53, 99, 48, 99, 97, 45, 57, 56, 98, 99, 45, 52, 48, 98, 52, 45, 56, 57, 50, 101, 45, 101, 51, 102, 53, 51, 98, 48, 56, 54, 98, 49, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c." peer_addr = [fd00:1122:3344:101::1b]:52206 req_id = 91048150-b88d-4ebb-bdbd-0a17977af2fc
1232026-07-29T16:45:01.280ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16154, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:43303 req_id = 58ec8d2a-f6a5-475b-a5a9-7238bbbfbde7
1242026-07-29T16:45:01.281ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16154, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:34492 req_id = 34e3272e-a685-486c-89d9-69efeb5eb295
1252026-07-29T16:45:01.379ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52151, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58431 req_id = 45fba28b-639e-4181-9b4e-f0f1b9ebd6cf