Max level shown:
1 [ Mar 26 16:26:36 Enabled. ]
2 [ Mar 26 16:26:36 Rereading configuration. ]
3 [ Mar 26 16:26:37 Rereading configuration. ]
4 [ Mar 26 16:26:38 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:1::1]:5353 --dns-address [fd00:1122:3344:1::1]:53 &"). ]
5 [ Mar 26 16:26:38 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-03-26T16:26:38.530ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:1::1]:53 }
92026-03-26T16:26:38.530ZINFOdns-server (store): opening sled database path = /data/dns
102026-03-26T16:26:38.553ZINFOdns-server (store): pruning trees for generations newer than 0
112026-03-26T16:26:38.553ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-03-26T16:26:38.553ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:1::1]:53
132026-03-26T16:26:38.553ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:1::1]:5353
142026-03-26T16:26:39.193ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:1::2]:33879
152026-03-26T16:26:39.194ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 712bdbb8-5a02-4d51-a765-88af15e0957c
162026-03-26T16:26:39.194ZINFOdns-server (store): pruning trees for generations newer than 0
172026-03-26T16:26:39.222ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-03-26T16:26:39.223ZINFOdns-server (store): updated generation new_generation = 1 req_id = 712bdbb8-5a02-4d51-a765-88af15e0957c
192026-03-26T16:26:39.223ZINFOdns-server (http): request completed latency_us = 29068 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:1::2]:33879 req_id = 712bdbb8-5a02-4d51-a765-88af15e0957c response_code = 204 uri = /config
202026-03-26T16:26:51.508ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63536, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:40677 req_id = 15299d14-ed42-4cd5-bcfe-5adffe52c626
212026-03-26T16:26:51.512ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15773, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49983 req_id = e566d2dd-64f3-4544-a1d3-fcb44c60c306
222026-03-26T16:26:51.516ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40775, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:54387 req_id = b9544a39-434f-4ee6-8b26-4d9eed5f9dd2
232026-03-26T16:28:15.536ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63916, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:43718 req_id = 969317cb-6f0f-419b-b9f0-60c25fd8ea8e
242026-03-26T16:28:15.609ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51172, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:42502 req_id = 4a1773c1-dc30-4f0c-83c2-e9ba2fe48b23
252026-03-26T16:28:15.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54924, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:36526 req_id = 665f8c85-7995-49bc-9c59-61260ae5533b
262026-03-26T16:28:28.354ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5842, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55193 req_id = cf64802b-7894-4505-b97d-6c1d9496277b
272026-03-26T16:28:31.609ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55614, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), ), original: Query { name: Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 98, 56, 55, 102, 55, 48, 101, 45, 50, 99, 53, 48, 45, 52, 54, 98, 51, 45, 57, 50, 48, 51, 45, 49, 54, 49, 48, 54, 51, 97, 101, 48, 57, 97, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab." peer_addr = [fd00:1122:3344:101::e]:42155 req_id = a8755af6-ecc0-499d-b236-6ea1d0c60aab
282026-03-26T16:28:31.610ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55614, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), ), original: Query { name: Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 98, 56, 55, 102, 55, 48, 101, 45, 50, 99, 53, 48, 45, 52, 54, 98, 51, 45, 57, 50, 48, 51, 45, 49, 54, 49, 48, 54, 51, 97, 101, 48, 57, 97, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab." peer_addr = [fd00:1122:3344:101::e]:35109 req_id = 0bd5055d-693d-40a6-a941-c895dcea8df3
292026-03-26T16:28:38.354ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10727, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:14902 req_id = 5b3069e6-fb6d-43d3-b99c-dd371cfb9272
302026-03-26T16:28:38.358ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8346, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64757 req_id = a817e88f-2906-4806-b678-ac6813956cfc
312026-03-26T16:28:38.706ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10128, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:14701 req_id = 4d35fa0e-05cd-487c-a5e9-0c5c891dce4c
322026-03-26T16:28:38.876ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51089, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:25693 req_id = 31582337-2743-49ff-b2ea-fd85a8dc9bee
332026-03-26T16:28:48.359ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30361, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61770 req_id = 7f0819d3-fc25-4c53-b0d5-c85495a1a44d
342026-03-26T16:28:56.848ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33360, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:40283 req_id = 2e8b3309-0973-4dcf-b530-58d6f19f98dd
352026-03-26T16:28:57.122ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:32951
362026-03-26T16:28:57.229ZINFOdns-server (http): request completed latency_us = 78988 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:32951 req_id = 923f56cd-ce6a-438d-973e-2eb61739a381 response_code = 200 uri = /config
372026-03-26T16:28:57.900ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:49747
382026-03-26T16:28:57.915ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 2ab09c3c-485a-40ef-92dd-d058caa52249
392026-03-26T16:28:57.915ZINFOdns-server (store): updated generation new_generation = 1 req_id = 2ab09c3c-485a-40ef-92dd-d058caa52249
402026-03-26T16:28:57.915ZINFOdns-server (http): request completed latency_us = 12711 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:49747 req_id = 2ab09c3c-485a-40ef-92dd-d058caa52249 response_code = 204 uri = /config
412026-03-26T16:28:57.922ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31377, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:40132 req_id = bd64fc11-79f9-43dc-9ce0-c112a14507bf
422026-03-26T16:28:58.040ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:46662
432026-03-26T16:28:58.040ZINFOdns-server (http): request completed latency_us = 214 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:46662 req_id = 78b51bcf-45f7-4001-a548-5935384fd209 response_code = 200 uri = /config
442026-03-26T16:28:58.361ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27493, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57147 req_id = 0bb6d180-c1c9-4ca9-96ae-5d6c5403d4c6
452026-03-26T16:28:59.533ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7516, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:13549 req_id = 8c8c148f-3344-42c3-8611-e39b477b0f4d
462026-03-26T16:28:59.534ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47268, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:27851 req_id = e0c24a98-9b8e-4fd4-aa49-8859957b664a
472026-03-26T16:28:59.536ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60629, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:39530 req_id = fe74497a-235a-4acb-80e9-ada7e292a09f
482026-03-26T16:28:59.702ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:32979
492026-03-26T16:28:59.702ZINFOdns-server (http): request completed latency_us = 192 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:32979 req_id = df6a46df-3fe9-44b7-bd13-80a8d09407bb response_code = 200 uri = /config
502026-03-26T16:28:59.763ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:57007
512026-03-26T16:28:59.763ZINFOdns-server (http): request completed latency_us = 234 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:57007 req_id = 11aae89d-a9de-49fd-8c82-bf1ae3ae3329 response_code = 200 uri = /config
522026-03-26T16:28:59.857ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:60940
532026-03-26T16:28:59.858ZINFOdns-server (http): request completed latency_us = 203 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:60940 req_id = 07c0abc3-4d81-482e-957f-4ff48ef59af2 response_code = 200 uri = /config
542026-03-26T16:29:08.362ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60378, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64840 req_id = 6b40ef22-38ca-49e6-ba42-9ea10947243d
552026-03-26T16:29:18.363ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26457, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56502 req_id = d9c4af94-268f-4935-acf3-69ec11cfbc29
562026-03-26T16:29:28.365ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57149, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50481 req_id = a3e20ad5-5b43-484b-b566-7a91d354f4d8
572026-03-26T16:29:38.285ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:46921
582026-03-26T16:29:38.287ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 54789360-2ebd-43f4-a312-c3cd5b9e5b0a
592026-03-26T16:29:38.287ZINFOdns-server (store): updated generation new_generation = 1 req_id = 54789360-2ebd-43f4-a312-c3cd5b9e5b0a
602026-03-26T16:29:38.287ZINFOdns-server (http): request completed latency_us = 221 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:46921 req_id = 54789360-2ebd-43f4-a312-c3cd5b9e5b0a response_code = 204 uri = /config
612026-03-26T16:29:38.367ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14576, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:65510 req_id = 03b35ee7-57cd-4d94-8397-018eef0c8b74
622026-03-26T16:29:38.685ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:40534
632026-03-26T16:29:38.687ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = a319d909-5827-4baa-8ecd-7cf2bd82d0de
642026-03-26T16:29:38.687ZINFOdns-server (store): updated generation new_generation = 1 req_id = a319d909-5827-4baa-8ecd-7cf2bd82d0de
652026-03-26T16:29:38.687ZINFOdns-server (http): request completed latency_us = 193 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:40534 req_id = a319d909-5827-4baa-8ecd-7cf2bd82d0de response_code = 204 uri = /config
662026-03-26T16:29:38.837ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:45748
672026-03-26T16:29:38.840ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 5b206d5d-9139-409c-a060-4b53b171be1d
682026-03-26T16:29:38.840ZINFOdns-server (store): updated generation new_generation = 1 req_id = 5b206d5d-9139-409c-a060-4b53b171be1d
692026-03-26T16:29:38.840ZINFOdns-server (http): request completed latency_us = 285 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:45748 req_id = 5b206d5d-9139-409c-a060-4b53b171be1d response_code = 204 uri = /config
702026-03-26T16:29:40.295ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57904, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:50949 req_id = aa05e00b-3f73-4772-816f-4001a59eca14
712026-03-26T16:29:40.295ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52299, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:41797 req_id = 50b38e6d-2dba-4528-a87d-cb82475e369d
722026-03-26T16:29:40.301ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5237, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:25899 req_id = f7b9fe4c-11b7-4002-af8c-aba78f97868a
732026-03-26T16:29:40.402ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::c]:62401
742026-03-26T16:29:40.402ZINFOdns-server (http): request completed latency_us = 198 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:62401 req_id = 3343ec25-d4f5-4984-b122-fec286760a89 response_code = 200 uri = /config
752026-03-26T16:29:40.437ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::b]:37721
762026-03-26T16:29:40.438ZINFOdns-server (http): request completed latency_us = 234 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:37721 req_id = 6d68c822-3248-45f2-a7cf-1dfd1a189b27 response_code = 200 uri = /config
772026-03-26T16:29:40.471ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:101::a]:62753
782026-03-26T16:29:40.471ZINFOdns-server (http): request completed latency_us = 151 local_addr = [fd00:1122:3344:1::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:62753 req_id = 6fcefb9d-f450-4d42-8c01-bbdb456cc434 response_code = 200 uri = /config
792026-03-26T16:29:48.368ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49318, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51847 req_id = 9b8ac9bf-4c4e-45db-996b-fc836bbc74d3
802026-03-26T16:29:58.370ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39727, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53855 req_id = a65d6f0b-bada-4c78-af2c-bd868ee7f48c
812026-03-26T16:30:00.137ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56642, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e88c2bcf-1329-4868-8248-f56ec76ec3d6."), ), original: Query { name: Name("oxz_cockroachdb_e88c2bcf-1329-4868-8248-f56ec76ec3d6."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 56, 56, 99, 50, 98, 99, 102, 45, 49, 51, 50, 57, 45, 52, 56, 54, 56, 45, 56, 50, 52, 56, 45, 102, 53, 54, 101, 99, 55, 54, 101, 99, 51, 100, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e88c2bcf-1329-4868-8248-f56ec76ec3d6." peer_addr = [fd00:1122:3344:101::4]:57035 req_id = 7f6a0798-4133-4483-9da4-f7f1b9909084
822026-03-26T16:30:00.138ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56642, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e88c2bcf-1329-4868-8248-f56ec76ec3d6."), ), original: Query { name: Name("oxz_cockroachdb_e88c2bcf-1329-4868-8248-f56ec76ec3d6."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 56, 56, 99, 50, 98, 99, 102, 45, 49, 51, 50, 57, 45, 52, 56, 54, 56, 45, 56, 50, 52, 56, 45, 102, 53, 54, 101, 99, 55, 54, 101, 99, 51, 100, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e88c2bcf-1329-4868-8248-f56ec76ec3d6." peer_addr = [fd00:1122:3344:101::4]:35994 req_id = 4a4d15fe-670c-4ac7-8fdf-05a6139292e2
832026-03-26T16:30:00.146ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24738, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7a6b0f69-c0bf-4c86-bdb4-9b340e64c374."), ), original: Query { name: Name("oxz_cockroachdb_7a6b0f69-c0bf-4c86-bdb4-9b340e64c374."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 97, 54, 98, 48, 102, 54, 57, 45, 99, 48, 98, 102, 45, 52, 99, 56, 54, 45, 98, 100, 98, 52, 45, 57, 98, 51, 52, 48, 101, 54, 52, 99, 51, 55, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7a6b0f69-c0bf-4c86-bdb4-9b340e64c374." peer_addr = [fd00:1122:3344:101::5]:33777 req_id = 57e41038-77fc-43ed-9aab-e42fbd72b11f
842026-03-26T16:30:00.146ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24738, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7a6b0f69-c0bf-4c86-bdb4-9b340e64c374."), ), original: Query { name: Name("oxz_cockroachdb_7a6b0f69-c0bf-4c86-bdb4-9b340e64c374."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 97, 54, 98, 48, 102, 54, 57, 45, 99, 48, 98, 102, 45, 52, 99, 56, 54, 45, 98, 100, 98, 52, 45, 57, 98, 51, 52, 48, 101, 54, 52, 99, 51, 55, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7a6b0f69-c0bf-4c86-bdb4-9b340e64c374." peer_addr = [fd00:1122:3344:101::5]:53630 req_id = 90c40f78-d474-4b6a-9818-257be6655ecb
852026-03-26T16:30:00.184ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40109, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b620acc9-004b-42d4-b805-8912dac3efc3."), ), original: Query { name: Name("oxz_cockroachdb_b620acc9-004b-42d4-b805-8912dac3efc3."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 54, 50, 48, 97, 99, 99, 57, 45, 48, 48, 52, 98, 45, 52, 50, 100, 52, 45, 98, 56, 48, 53, 45, 56, 57, 49, 50, 100, 97, 99, 51, 101, 102, 99, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b620acc9-004b-42d4-b805-8912dac3efc3." peer_addr = [fd00:1122:3344:101::3]:32938 req_id = 1855821c-0ec3-45d9-84e5-a896589883b5
862026-03-26T16:30:00.185ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40109, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b620acc9-004b-42d4-b805-8912dac3efc3."), ), original: Query { name: Name("oxz_cockroachdb_b620acc9-004b-42d4-b805-8912dac3efc3."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 54, 50, 48, 97, 99, 99, 57, 45, 48, 48, 52, 98, 45, 52, 50, 100, 52, 45, 98, 56, 48, 53, 45, 56, 57, 49, 50, 100, 97, 99, 51, 101, 102, 99, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b620acc9-004b-42d4-b805-8912dac3efc3." peer_addr = [fd00:1122:3344:101::3]:35476 req_id = a23a828a-2ca5-4b72-962d-78e089d6a5ed
872026-03-26T16:30:00.195ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19352, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ce21c424-3731-45c4-8a11-2c5065a3a90c."), ), original: Query { name: Name("oxz_cockroachdb_ce21c424-3731-45c4-8a11-2c5065a3a90c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 101, 50, 49, 99, 52, 50, 52, 45, 51, 55, 51, 49, 45, 52, 53, 99, 52, 45, 56, 97, 49, 49, 45, 50, 99, 53, 48, 54, 53, 97, 51, 97, 57, 48, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ce21c424-3731-45c4-8a11-2c5065a3a90c." peer_addr = [fd00:1122:3344:101::6]:56322 req_id = 60b2ed48-48b9-4534-850c-d0f0c5461ca8
882026-03-26T16:30:00.196ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19352, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ce21c424-3731-45c4-8a11-2c5065a3a90c."), ), original: Query { name: Name("oxz_cockroachdb_ce21c424-3731-45c4-8a11-2c5065a3a90c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 101, 50, 49, 99, 52, 50, 52, 45, 51, 55, 51, 49, 45, 52, 53, 99, 52, 45, 56, 97, 49, 49, 45, 50, 99, 53, 48, 54, 53, 97, 51, 97, 57, 48, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ce21c424-3731-45c4-8a11-2c5065a3a90c." peer_addr = [fd00:1122:3344:101::6]:38271 req_id = 230c3e65-3af4-4992-8d72-ea4a2f45abff
892026-03-26T16:30:00.234ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59899, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_75629e5e-4d5f-463e-b74b-33eb84495400."), ), original: Query { name: Name("oxz_cockroachdb_75629e5e-4d5f-463e-b74b-33eb84495400."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 53, 54, 50, 57, 101, 53, 101, 45, 52, 100, 53, 102, 45, 52, 54, 51, 101, 45, 98, 55, 52, 98, 45, 51, 51, 101, 98, 56, 52, 52, 57, 53, 52, 48, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_75629e5e-4d5f-463e-b74b-33eb84495400." peer_addr = [fd00:1122:3344:101::7]:50231 req_id = fb152ce3-bea6-4d4f-9a16-2e26cce52911
902026-03-26T16:30:00.234ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59899, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_75629e5e-4d5f-463e-b74b-33eb84495400."), ), original: Query { name: Name("oxz_cockroachdb_75629e5e-4d5f-463e-b74b-33eb84495400."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 53, 54, 50, 57, 101, 53, 101, 45, 52, 100, 53, 102, 45, 52, 54, 51, 101, 45, 98, 55, 52, 98, 45, 51, 51, 101, 98, 56, 52, 52, 57, 53, 52, 48, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_75629e5e-4d5f-463e-b74b-33eb84495400." peer_addr = [fd00:1122:3344:101::7]:54939 req_id = eee974e9-3d6e-4920-8fbe-2ae9cdc0da60
912026-03-26T16:30:00.516ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13188, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), ), original: Query { name: Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 98, 56, 55, 102, 55, 48, 101, 45, 50, 99, 53, 48, 45, 52, 54, 98, 51, 45, 57, 50, 48, 51, 45, 49, 54, 49, 48, 54, 51, 97, 101, 48, 57, 97, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab." peer_addr = [fd00:1122:3344:101::e]:47197 req_id = 58663cbe-f60e-4b3c-b07b-d137faa0967f
922026-03-26T16:30:00.517ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13188, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), ), original: Query { name: Name("oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 98, 56, 55, 102, 55, 48, 101, 45, 50, 99, 53, 48, 45, 52, 54, 98, 51, 45, 57, 50, 48, 51, 45, 49, 54, 49, 48, 54, 51, 97, 101, 48, 57, 97, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6b87f70e-2c50-46b3-9203-161063ae09ab." peer_addr = [fd00:1122:3344:101::e]:35781 req_id = 44f584ca-97e0-4cb3-9ae8-f1178d1c6af1