Max level shown:
1 [ Jul 29 16:39:06 Enabled. ]
2 [ Jul 29 16:39:06 Rereading configuration. ]
3 [ Jul 29 16:39:07 Rereading configuration. ]
4 [ Jul 29 16:39:08 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:1::1]:5353 --dns-address [fd00:1122:3344:1::1]:53 &"). ]
5 [ Jul 29 16:39:08 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-29T16:39:08.217ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:1::1]:53 }
92026-07-29T16:39:08.218ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-29T16:39:08.239ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-29T16:39:08.239ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-29T16:39:08.239ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:1::1]:53
132026-07-29T16:39:08.239ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:1::1]:5353
142026-07-29T16:39:08.358ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:1::2]:62407
152026-07-29T16:39:08.360ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 586defad-574c-453c-b2f1-c4c26125e0e9
162026-07-29T16:39:08.360ZINFOdns-server (store): pruning trees for generations newer than 0
172026-07-29T16:39:08.372ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-07-29T16:39:08.372ZINFOdns-server (store): updated generation new_generation = 1 req_id = 586defad-574c-453c-b2f1-c4c26125e0e9
192026-07-29T16:39:08.372ZINFOdns-server (http): request completed latency_us = 13103 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:1::2]:62407 req_id = 586defad-574c-453c-b2f1-c4c26125e0e9 response_code = 204 uri = /config
202026-07-29T16:39:20.696ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57074, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:64374 req_id = c2fd657d-133c-4e2b-b397-de2cfc914ec4
212026-07-29T16:39:20.700ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50324, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:38747 req_id = 2b3c4e27-b9e5-44b1-be5e-2935efa01f82
222026-07-29T16:39:20.704ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56411, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:50916 req_id = 7fcc98bd-b831-45ef-ba00-b5c04fba8adf
232026-07-29T16:41:20.693ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39323, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:44433 req_id = 2eb51e4f-0d98-402e-8d62-03704e2376ab
242026-07-29T16:41:20.698ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63584, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49763 req_id = c0c7d96a-e8bb-430b-af06-f74eb05d2811
252026-07-29T16:41:20.703ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:37695 req_id = 8288a068-8b43-4b4b-be74-29b52dcfc37f
262026-07-29T16:42:41.385ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6129, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50838 req_id = ed2a9159-1369-4be9-8a00-a141ccffe1fd
272026-07-29T16:42:51.357ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14964, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49271 req_id = 7418c6e8-e8dd-4c2d-8fbe-0ffaa29bcc07
282026-07-29T16:42:54.390ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57173, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:18437 req_id = d9ae2ef7-a0c5-42fc-8636-34dd56b5a1a1
292026-07-29T16:42:55.233ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5886, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:12941 req_id = 13de81c0-5f10-44a0-9fee-4f7517e62363
302026-07-29T16:42:55.472ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53740, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:42827 req_id = ccdb7bb9-b6b4-4613-b174-21c94e53b9e5
312026-07-29T16:42:56.594ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:42128
322026-07-29T16:42:56.599ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:39660
332026-07-29T16:42:56.648ZINFOdns-server (http): request completed latency_us = 30011 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:39660 req_id = 4e2c45ba-f65f-406f-b885-37214864b8a8 response_code = 200 uri = /config
342026-07-29T16:42:56.648ZINFOdns-server (http): request completed latency_us = 30020 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:42128 req_id = 2b8e2c8e-6f5f-40d3-bb1a-ff1268447737 response_code = 200 uri = /config
352026-07-29T16:42:57.315ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30949, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:63469 req_id = 3ed2321b-69d5-46cf-b878-6870a7ba97d4
362026-07-29T16:42:57.324ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30949, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:49654 req_id = 39e35820-941e-4d44-8312-0726c9719613
372026-07-29T16:42:57.998ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43635, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:2188 req_id = 4db6c9a8-8177-42b2-b772-c6a2776457f8
382026-07-29T16:42:58.224ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:47413
392026-07-29T16:42:58.224ZINFOdns-server (http): request completed latency_us = 250 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:47413 req_id = e0699044-ab81-4295-96b6-662e1bdc4e49 response_code = 200 uri = /config
402026-07-29T16:42:59.159ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25222, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:30294 req_id = 9ada804f-42bd-4f64-a921-e91b5348f1c3
412026-07-29T16:42:59.241ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:36600
422026-07-29T16:42:59.243ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 995a683c-7708-4680-905e-8520d7364099
432026-07-29T16:42:59.244ZINFOdns-server (store): updated generation new_generation = 1 req_id = 995a683c-7708-4680-905e-8520d7364099
442026-07-29T16:42:59.244ZINFOdns-server (http): request completed latency_us = 380 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:36600 req_id = 995a683c-7708-4680-905e-8520d7364099 response_code = 204 uri = /config
452026-07-29T16:42:59.297ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:50715
462026-07-29T16:42:59.298ZINFOdns-server (http): request completed latency_us = 1122 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:50715 req_id = e3a0addd-dfbb-4fed-9fe3-a8ad187db211 response_code = 200 uri = /config
472026-07-29T16:43:00.468ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55780, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:35966 req_id = 13017818-5e0f-4fbc-8230-035d47c36e16
482026-07-29T16:43:00.593ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:53448
492026-07-29T16:43:00.593ZINFOdns-server (http): request completed latency_us = 350 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:53448 req_id = c3ffe883-09b8-48e9-ab24-fd363671c0f7 response_code = 200 uri = /config
502026-07-29T16:43:01.358ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54178, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55947 req_id = c4ffad4d-7850-47a8-bf36-55ef2fee2ccf
512026-07-29T16:43:05.523ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19490, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:56842 req_id = e31068be-b096-463d-b346-1a50b874bb50
522026-07-29T16:43:05.820ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:44751
532026-07-29T16:43:05.821ZINFOdns-server (http): request completed latency_us = 270 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:44751 req_id = c9528de1-f010-49cd-b7b5-7962f211f254 response_code = 200 uri = /config
542026-07-29T16:43:06.404ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30216, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:43262 req_id = f0f14d40-f791-499d-aa02-56d43ee6b9ac
552026-07-29T16:43:06.515ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:53810
562026-07-29T16:43:06.515ZINFOdns-server (http): request completed latency_us = 211 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:53810 req_id = 2d95c6f3-0876-4056-abe5-95c5a07c2a59 response_code = 200 uri = /config
572026-07-29T16:43:11.361ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50636, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52823 req_id = 290f8e10-1315-476e-90be-16ba6e968124
582026-07-29T16:43:21.362ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50690, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61106 req_id = 91828bb6-4b9b-4c23-824f-29bc08abfada
592026-07-29T16:43:31.363ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30563, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55335 req_id = 8afd8a14-ae23-46d0-9da9-ff29c7e99dfa
602026-07-29T16:43:41.366ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9518, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56627 req_id = e2dc26cd-3c3d-4cef-af10-a48ca34e6c6e
612026-07-29T16:43:51.367ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39048, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57152 req_id = 07b6ad90-35be-420d-9507-f792e15f4cb4
622026-07-29T16:43:54.206ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:50394
632026-07-29T16:43:54.210ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = c4efd150-e996-40a2-8984-f2b30f9b2b1a
642026-07-29T16:43:54.210ZINFOdns-server (store): updated generation new_generation = 1 req_id = c4efd150-e996-40a2-8984-f2b30f9b2b1a
652026-07-29T16:43:54.210ZINFOdns-server (http): request completed latency_us = 313 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:50394 req_id = c4efd150-e996-40a2-8984-f2b30f9b2b1a response_code = 204 uri = /config
662026-07-29T16:43:55.100ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:60890
672026-07-29T16:43:55.104ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = c6673cff-49ed-4698-9d88-347bfad2fa87
682026-07-29T16:43:55.104ZINFOdns-server (store): updated generation new_generation = 1 req_id = c6673cff-49ed-4698-9d88-347bfad2fa87
692026-07-29T16:43:55.104ZINFOdns-server (http): request completed latency_us = 269 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:60890 req_id = c6673cff-49ed-4698-9d88-347bfad2fa87 response_code = 204 uri = /config
702026-07-29T16:43:55.436ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:38102
712026-07-29T16:43:55.439ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = a8ff71c8-1d44-413e-8765-c53e5a5fa0ad
722026-07-29T16:43:55.439ZINFOdns-server (store): updated generation new_generation = 1 req_id = a8ff71c8-1d44-413e-8765-c53e5a5fa0ad
732026-07-29T16:43:55.439ZINFOdns-server (http): request completed latency_us = 331 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:38102 req_id = a8ff71c8-1d44-413e-8765-c53e5a5fa0ad response_code = 204 uri = /config
742026-07-29T16:43:55.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3499, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:46719 req_id = afaa2739-756f-44e8-bff3-52298385fb19
752026-07-29T16:43:55.744ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:38192
762026-07-29T16:43:55.745ZINFOdns-server (http): request completed latency_us = 211 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:38192 req_id = ab31c173-9fcb-4984-a8a3-6ce5a030f90a response_code = 200 uri = /config
772026-07-29T16:43:56.154ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60713, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:18800 req_id = 1f9a4c75-7179-4055-a8dc-95fac6c0b4d9
782026-07-29T16:43:56.257ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:39522
792026-07-29T16:43:56.257ZINFOdns-server (http): request completed latency_us = 229 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:39522 req_id = d5363863-da30-48bc-bf50-694683b2b698 response_code = 200 uri = /config
802026-07-29T16:43:56.579ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19090, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:61765 req_id = 49b53db7-4a7b-44ae-976f-ac724ac5b442
812026-07-29T16:43:56.772ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:32932
822026-07-29T16:43:56.772ZINFOdns-server (http): request completed latency_us = 244 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:32932 req_id = cc2c60ca-496a-4195-a277-a3fc734a36a4 response_code = 200 uri = /config
832026-07-29T16:44:01.369ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9125, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54872 req_id = 26270578-10fc-4e8d-9880-6fc883858e08
842026-07-29T16:44:11.370ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7952, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64608 req_id = c9dcc499-6125-4783-a738-18c116adb2bf
852026-07-29T16:44:21.372ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63106, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60193 req_id = e4104bdc-d9b0-47be-ac65-30f1d05f2a87
862026-07-29T16:44:31.374ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52222, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57712 req_id = a287e06c-8d26-4c48-958e-63b0f8b24efb
872026-07-29T16:44:41.376ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17534, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54369 req_id = a428ee3c-6107-4c58-b43a-f563a0cc938e
882026-07-29T16:44:51.378ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45735, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49348 req_id = 298c39cc-ce81-4886-a958-1c176aa86e9e
892026-07-29T16:44:54.123ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:47384
902026-07-29T16:44:54.125ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = a11ada9e-467e-4e91-8333-14bb96c1b7b4
912026-07-29T16:44:54.125ZINFOdns-server (store): updated generation new_generation = 1 req_id = a11ada9e-467e-4e91-8333-14bb96c1b7b4
922026-07-29T16:44:54.125ZINFOdns-server (http): request completed latency_us = 238 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:47384 req_id = a11ada9e-467e-4e91-8333-14bb96c1b7b4 response_code = 204 uri = /config
932026-07-29T16:44:54.895ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2683, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:24975 req_id = fab1d699-13e8-4a64-ad06-9acd8739971a
942026-07-29T16:44:54.993ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:46543
952026-07-29T16:44:54.994ZINFOdns-server (http): request completed latency_us = 204 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:46543 req_id = a723f3e5-d488-4fdf-b97b-782e94392c5f response_code = 200 uri = /config
962026-07-29T16:44:55.100ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:59751
972026-07-29T16:44:55.103ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = c6c0a7fe-f6bc-4b22-812e-b396f90deb6b
982026-07-29T16:44:55.103ZINFOdns-server (store): updated generation new_generation = 1 req_id = c6c0a7fe-f6bc-4b22-812e-b396f90deb6b
992026-07-29T16:44:55.103ZINFOdns-server (http): request completed latency_us = 250 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:59751 req_id = c6c0a7fe-f6bc-4b22-812e-b396f90deb6b response_code = 204 uri = /config
1002026-07-29T16:44:55.198ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:43279
1012026-07-29T16:44:55.201ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 0f97fead-9fda-4d9b-b386-839843f7e49a
1022026-07-29T16:44:55.201ZINFOdns-server (store): updated generation new_generation = 1 req_id = 0f97fead-9fda-4d9b-b386-839843f7e49a
1032026-07-29T16:44:55.201ZINFOdns-server (http): request completed latency_us = 271 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:43279 req_id = 0f97fead-9fda-4d9b-b386-839843f7e49a response_code = 204 uri = /config
1042026-07-29T16:44:56.395ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27617, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:5108 req_id = 58e90503-daf2-4852-a02b-41f614e2a297
1052026-07-29T16:44:56.400ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58695, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:24209 req_id = 63100580-6467-4ea7-a6c3-c3681e342eb5
1062026-07-29T16:44:56.501ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:42616
1072026-07-29T16:44:56.501ZINFOdns-server (http): request completed latency_us = 270 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:42616 req_id = 3f9e2322-1f8f-4796-bd48-08db0b67443c response_code = 200 uri = /config
1082026-07-29T16:44:56.591ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:37585
1092026-07-29T16:44:56.591ZINFOdns-server (http): request completed latency_us = 208 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:37585 req_id = 23e55c98-68c3-45b4-ae9a-81be6605cc76 response_code = 200 uri = /config
1102026-07-29T16:45:00.034ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), ), original: Query { name: Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 49, 57, 57, 51, 55, 54, 102, 45, 57, 102, 98, 101, 45, 52, 101, 56, 100, 45, 97, 99, 48, 102, 45, 50, 99, 49, 50, 49, 100, 48, 51, 54, 55, 54, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e." peer_addr = [fd00:1122:3344:101::3]:43073 req_id = 8ca09941-f716-430e-8f1d-e8a85a2c3041
1112026-07-29T16:45:00.035ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), ), original: Query { name: Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 49, 57, 57, 51, 55, 54, 102, 45, 57, 102, 98, 101, 45, 52, 101, 56, 100, 45, 97, 99, 48, 102, 45, 50, 99, 49, 50, 49, 100, 48, 51, 54, 55, 54, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e." peer_addr = [fd00:1122:3344:101::3]:65370 req_id = 4eea056a-c0f7-46b5-bc64-4d77eb88ea26
1122026-07-29T16:45:00.086ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54786, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), ), original: Query { name: Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 56, 56, 50, 102, 97, 102, 98, 45, 54, 98, 51, 49, 45, 52, 101, 98, 55, 45, 97, 99, 54, 50, 45, 52, 50, 53, 53, 97, 101, 100, 51, 99, 48, 56, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c." peer_addr = [fd00:1122:3344:101::5]:33058 req_id = 35381174-8a7c-4fad-af40-f0d8479aa046
1132026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), ), original: Query { name: Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 56, 57, 97, 51, 53, 98, 57, 45, 101, 98, 54, 101, 45, 52, 48, 56, 97, 45, 97, 48, 49, 97, 45, 101, 57, 56, 51, 57, 97, 98, 51, 52, 99, 97, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8." peer_addr = [fd00:1122:3344:101::4]:52011 req_id = 2f51ffbb-acc7-413d-8525-f3b6fc6168a5
1142026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54786, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), ), original: Query { name: Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 56, 56, 50, 102, 97, 102, 98, 45, 54, 98, 51, 49, 45, 52, 101, 98, 55, 45, 97, 99, 54, 50, 45, 52, 50, 53, 53, 97, 101, 100, 51, 99, 48, 56, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c." peer_addr = [fd00:1122:3344:101::5]:60018 req_id = cc1dd9ea-2f71-4141-8370-49ccfca489b4
1152026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), ), original: Query { name: Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 56, 57, 97, 51, 53, 98, 57, 45, 101, 98, 54, 101, 45, 52, 48, 56, 97, 45, 97, 48, 49, 97, 45, 101, 57, 56, 51, 57, 97, 98, 51, 52, 99, 97, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8." peer_addr = [fd00:1122:3344:101::4]:52017 req_id = c9cd1fb7-3b33-4178-8737-4ebc4d8f6554
1162026-07-29T16:45:00.089ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), ), original: Query { name: Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 53, 98, 54, 52, 54, 48, 53, 45, 51, 57, 48, 49, 45, 52, 48, 100, 99, 45, 98, 48, 57, 48, 45, 56, 99, 49, 48, 56, 54, 56, 53, 57, 49, 102, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff." peer_addr = [fd00:1122:3344:101::7]:48817 req_id = 3d3fcf6b-c09a-415c-a6e3-dc5799e677d7
1172026-07-29T16:45:00.090ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), ), original: Query { name: Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 53, 98, 54, 52, 54, 48, 53, 45, 51, 57, 48, 49, 45, 52, 48, 100, 99, 45, 98, 48, 57, 48, 45, 56, 99, 49, 48, 56, 54, 56, 53, 57, 49, 102, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff." peer_addr = [fd00:1122:3344:101::7]:43365 req_id = 1247ddf3-c609-4865-96c7-db30542a49d3
1182026-07-29T16:45:00.633ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:34815 req_id = 9b4e5855-7270-4b82-8f92-2e351a1520fc
1192026-07-29T16:45:00.633ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:63791 req_id = c5663fa8-dd0d-43fc-98da-9e5454bc296f
1202026-07-29T16:45:01.012ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), ), original: Query { name: Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 102, 57, 53, 55, 101, 98, 101, 45, 53, 101, 52, 57, 45, 52, 57, 55, 55, 45, 97, 52, 50, 101, 45, 49, 53, 51, 52, 54, 52, 98, 51, 99, 56, 52, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847." peer_addr = [fd00:1122:3344:101::6]:61311 req_id = ae3987e1-4ab3-4169-b191-6e20415eaea3
1212026-07-29T16:45:01.013ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), ), original: Query { name: Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 102, 57, 53, 55, 101, 98, 101, 45, 53, 101, 52, 57, 45, 52, 57, 55, 55, 45, 97, 52, 50, 101, 45, 49, 53, 51, 52, 54, 52, 98, 51, 99, 56, 52, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847." peer_addr = [fd00:1122:3344:101::6]:63430 req_id = 5fe680ef-ebd9-4168-b850-bf33baa87d3d
1222026-07-29T16:45:01.110ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11174, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c."), ), original: Query { name: Name("oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 52, 53, 97, 53, 99, 48, 99, 97, 45, 57, 56, 98, 99, 45, 52, 48, 98, 52, 45, 56, 57, 50, 101, 45, 101, 51, 102, 53, 51, 98, 48, 56, 54, 98, 49, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c." peer_addr = [fd00:1122:3344:101::1b]:37652 req_id = 04e2716f-5e45-48a8-8cdd-3ee732013f7d
1232026-07-29T16:45:01.280ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16154, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:49633 req_id = b6644aaf-0a5d-4383-a63e-8be7a5b662c6
1242026-07-29T16:45:01.281ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16154, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:34275 req_id = 903a8d8f-74be-4a54-bbfb-6cc0f24dc20f
1252026-07-29T16:45:01.379ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55747, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50018 req_id = 03b4b001-3763-4923-924b-7412d7f12c96