Max level shown:
1 [ Apr 17 23:26:23 Enabled. ]
2 [ Apr 17 23:26:23 Rereading configuration. ]
3 [ Apr 17 23:26:24 Rereading configuration. ]
4 [ Apr 17 23:26:25 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Apr 17 23:26:25 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-04-17T23:26:25.480ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-04-17T23:26:25.481ZINFOdns-server (store): opening sled database path = /data/dns
102026-04-17T23:26:25.500ZINFOdns-server (store): pruning trees for generations newer than 0
112026-04-17T23:26:25.500ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-04-17T23:26:25.500ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-04-17T23:26:25.500ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-04-17T23:26:26.273ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:49783
152026-04-17T23:26:26.274ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 8a8a39b4-1b5d-4066-848a-982f6a9290ed
162026-04-17T23:26:26.274ZINFOdns-server (store): pruning trees for generations newer than 0
172026-04-17T23:26:26.290ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-04-17T23:26:26.290ZINFOdns-server (store): updated generation new_generation = 1 req_id = 8a8a39b4-1b5d-4066-848a-982f6a9290ed
192026-04-17T23:26:26.290ZINFOdns-server (http): request completed latency_us = 16855 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:49783 req_id = 8a8a39b4-1b5d-4066-848a-982f6a9290ed response_code = 204 uri = /config
202026-04-17T23:26:38.755ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32439, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:57572 req_id = aeb83d7d-58b3-49f0-ac2f-15dd53f2a6af
212026-04-17T23:26:38.760ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18032, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:58468 req_id = 808a1d17-5b4f-4ee3-8798-976851cedf4d
222026-04-17T23:26:38.764ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4463, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:60485 req_id = 19a9a6b8-2f0a-40e6-83f5-a6f8e5379edd
232026-04-17T23:28:28.016ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6616, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58683 req_id = 3437eec1-acd0-4501-a139-36223d63111b
242026-04-17T23:28:31.867ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:37818 req_id = 24634439-7f35-4cbe-892a-a7e7ff28a230
252026-04-17T23:28:31.873ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:46246 req_id = c7b69a63-9bc1-4cfc-a468-aed9c314b11c
262026-04-17T23:28:38.002ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50151, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60596 req_id = 42b9257f-4f25-4c69-8a35-2e285535e6d1
272026-04-17T23:28:38.195ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56752, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:53827 req_id = 08ca1d8a-4786-42c7-8420-d2256bd2c314
282026-04-17T23:28:38.197ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63818, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:47418 req_id = c6d0bb69-8ed3-4a8c-96fc-ff04ee74b2d6
292026-04-17T23:28:38.199ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34489, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:51797 req_id = 61442364-4a13-42e4-94b4-7040cfdf408a
302026-04-17T23:28:45.826ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26722, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:62905 req_id = e8825f92-dc86-4bc7-90d0-52c9c87a9a0b
312026-04-17T23:28:46.061ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:33088
322026-04-17T23:28:46.111ZINFOdns-server (http): request completed latency_us = 26770 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:33088 req_id = 5639ba75-e98a-4803-b3a1-4e4b903314de response_code = 200 uri = /config
332026-04-17T23:28:48.005ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45538, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52655 req_id = 23d7195c-ec01-4bbe-8c47-f1ba00d1477b
342026-04-17T23:28:48.167ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:53951
352026-04-17T23:28:48.174ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 749f64d1-255b-42c3-9a45-418c2329d622
362026-04-17T23:28:48.174ZINFOdns-server (store): updated generation new_generation = 1 req_id = 749f64d1-255b-42c3-9a45-418c2329d622
372026-04-17T23:28:48.174ZINFOdns-server (http): request completed latency_us = 4120 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:53951 req_id = 749f64d1-255b-42c3-9a45-418c2329d622 response_code = 204 uri = /config
382026-04-17T23:28:48.198ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9263, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:20930 req_id = 1d72078c-9b3f-443f-a43a-abaa715e2ab7
392026-04-17T23:28:48.438ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:34426
402026-04-17T23:28:48.440ZINFOdns-server (http): request completed latency_us = 801 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:34426 req_id = 8bb86502-e599-4007-92b4-47334ce79ca1 response_code = 200 uri = /config
412026-04-17T23:28:49.197ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39977, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:52822 req_id = a4413fbf-03d0-4026-ba00-24498efcfffa
422026-04-17T23:28:49.198ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28390, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:12082 req_id = 7a52195f-8bb1-48a4-8ed0-1a613c5e9993
432026-04-17T23:28:49.346ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:61140
442026-04-17T23:28:49.346ZINFOdns-server (http): request completed latency_us = 273 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:61140 req_id = ce89a3a9-2ef9-43f5-bcc7-317bbec9f7a8 response_code = 200 uri = /config
452026-04-17T23:28:49.445ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:39490
462026-04-17T23:28:49.446ZINFOdns-server (http): request completed latency_us = 271 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:39490 req_id = 3bcc7d93-bbe7-4a29-89eb-294a6247a79a response_code = 200 uri = /config
472026-04-17T23:28:50.785ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26815, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:33368 req_id = d0872dd1-7c4e-4947-9b04-6f72f4103413
482026-04-17T23:28:50.999ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:59097
492026-04-17T23:28:51.000ZINFOdns-server (http): request completed latency_us = 204 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:59097 req_id = ecaa892d-e36a-4397-9e11-fa4353c113f5 response_code = 200 uri = /config
502026-04-17T23:28:58.007ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37264, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50563 req_id = fe0f9596-a8a6-4628-a163-63b180a40544
512026-04-17T23:29:08.010ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17863, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62826 req_id = dddb5eec-2c41-446c-ba6a-000860828a60
522026-04-17T23:29:18.011ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33785, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57171 req_id = 899b95f4-11c8-439c-87f6-53d2ae2e29d8
532026-04-17T23:29:28.012ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 312, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56121 req_id = f5c4cbda-ed57-46d0-bcda-6daaaa27c965
542026-04-17T23:29:37.858ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:40991
552026-04-17T23:29:37.861ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 8b6d8129-583f-4924-83ba-bdbb6d5bc7ed
562026-04-17T23:29:37.861ZINFOdns-server (store): updated generation new_generation = 1 req_id = 8b6d8129-583f-4924-83ba-bdbb6d5bc7ed
572026-04-17T23:29:37.861ZINFOdns-server (http): request completed latency_us = 387 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:40991 req_id = 8b6d8129-583f-4924-83ba-bdbb6d5bc7ed response_code = 204 uri = /config
582026-04-17T23:29:38.014ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4281, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59560 req_id = 1da68f1b-9630-49c7-ace9-363d6a628190
592026-04-17T23:29:38.082ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:55917
602026-04-17T23:29:38.085ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 4b31f3fe-d270-472b-8b5b-3d5bf9173a4b
612026-04-17T23:29:38.085ZINFOdns-server (store): updated generation new_generation = 1 req_id = 4b31f3fe-d270-472b-8b5b-3d5bf9173a4b
622026-04-17T23:29:38.085ZINFOdns-server (http): request completed latency_us = 226 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:55917 req_id = 4b31f3fe-d270-472b-8b5b-3d5bf9173a4b response_code = 204 uri = /config
632026-04-17T23:29:38.275ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:41872
642026-04-17T23:29:38.278ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 5619130a-810e-4736-8b08-fcaecea2b48a
652026-04-17T23:29:38.278ZINFOdns-server (store): updated generation new_generation = 1 req_id = 5619130a-810e-4736-8b08-fcaecea2b48a
662026-04-17T23:29:38.278ZINFOdns-server (http): request completed latency_us = 295 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:41872 req_id = 5619130a-810e-4736-8b08-fcaecea2b48a response_code = 204 uri = /config
672026-04-17T23:29:39.249ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63215, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:43469 req_id = 5cdd24ec-145a-442b-8303-3dd1fab860fb
682026-04-17T23:29:39.309ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54063, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:37387 req_id = d89fffad-d292-4bb9-a49b-f34658679b37
692026-04-17T23:29:39.348ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:45125
702026-04-17T23:29:39.349ZINFOdns-server (http): request completed latency_us = 281 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:45125 req_id = 7dd34781-94b9-4fbf-9346-62f1fcb8c909 response_code = 200 uri = /config
712026-04-17T23:29:39.493ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:44191
722026-04-17T23:29:39.493ZINFOdns-server (http): request completed latency_us = 243 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:44191 req_id = cff2d8da-c022-4aad-b6a9-6f187d0acfcd response_code = 200 uri = /config
732026-04-17T23:29:40.049ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34210, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:53873 req_id = 2447b8b1-d2a6-46ca-bd3c-4b42474fa47e
742026-04-17T23:29:40.233ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:47338
752026-04-17T23:29:40.233ZINFOdns-server (http): request completed latency_us = 288 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:47338 req_id = d3ef3778-5ce9-4e6f-8793-c55727a4f868 response_code = 200 uri = /config
762026-04-17T23:29:48.016ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18039, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63324 req_id = 0ec074aa-6819-4ccf-a2d7-788918415b8f
772026-04-17T23:29:58.018ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49033, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64234 req_id = 98c36f1f-23ce-474d-bc73-79f2db16ad03
782026-04-17T23:30:00.206ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36603, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:52781 req_id = f2a100d2-770a-4014-a84d-81fdd69e836a
792026-04-17T23:30:00.206ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36603, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:44436 req_id = 99eefcaf-93c8-45fb-b4aa-99e2a2ce4063
802026-04-17T23:30:00.564ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), ), original: Query { name: Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 101, 98, 102, 99, 51, 52, 53, 45, 50, 55, 51, 49, 45, 52, 100, 100, 50, 45, 97, 98, 99, 53, 45, 51, 57, 55, 98, 98, 55, 52, 100, 53, 54, 50, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b." peer_addr = [fd00:1122:3344:101::3]:50738 req_id = aca75e4f-51fe-4102-ab54-d8eea30b5db0
812026-04-17T23:30:00.564ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), ), original: Query { name: Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 101, 98, 102, 99, 51, 52, 53, 45, 50, 55, 51, 49, 45, 52, 100, 100, 50, 45, 97, 98, 99, 53, 45, 51, 57, 55, 98, 98, 55, 52, 100, 53, 54, 50, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b." peer_addr = [fd00:1122:3344:101::3]:61235 req_id = f6276e87-2665-4ea0-87aa-811bbd19acc7
822026-04-17T23:30:00.569ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), ), original: Query { name: Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 50, 98, 53, 57, 54, 101, 56, 45, 100, 49, 99, 56, 45, 52, 57, 100, 57, 45, 57, 52, 48, 102, 45, 49, 48, 98, 52, 50, 50, 101, 53, 102, 101, 56, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80." peer_addr = [fd00:1122:3344:101::5]:53047 req_id = 1a11d3ea-476e-4ef8-b892-c35536f37f2e
832026-04-17T23:30:00.570ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), ), original: Query { name: Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 50, 98, 53, 57, 54, 101, 56, 45, 100, 49, 99, 56, 45, 52, 57, 100, 57, 45, 57, 52, 48, 102, 45, 49, 48, 98, 52, 50, 50, 101, 53, 102, 101, 56, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80." peer_addr = [fd00:1122:3344:101::5]:38867 req_id = 3b3871bc-63a2-4ffe-b732-05233937fd1c
842026-04-17T23:30:00.596ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43303, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:51686 req_id = b5a061b3-0909-49d5-b514-ceafec53c84c
852026-04-17T23:30:00.597ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43303, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:55494 req_id = 60bf8e32-97d4-4d36-b3ab-2c2472167219
862026-04-17T23:30:00.630ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22641, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), ), original: Query { name: Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 57, 55, 49, 101, 48, 51, 55, 45, 56, 57, 99, 99, 45, 52, 50, 99, 52, 45, 56, 53, 101, 102, 45, 102, 48, 97, 50, 54, 97, 56, 54, 51, 57, 53, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954." peer_addr = [fd00:1122:3344:101::6]:55470 req_id = 6a5b28c3-12ae-4acf-b159-2d40ce516856
872026-04-17T23:30:00.631ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22641, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), ), original: Query { name: Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 57, 55, 49, 101, 48, 51, 55, 45, 56, 57, 99, 99, 45, 52, 50, 99, 52, 45, 56, 53, 101, 102, 45, 102, 48, 97, 50, 54, 97, 56, 54, 51, 57, 53, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954." peer_addr = [fd00:1122:3344:101::6]:45178 req_id = 347a6749-b63e-481a-9451-e268461ae0fb
882026-04-17T23:30:00.651ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56236, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), ), original: Query { name: Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 54, 102, 57, 50, 48, 100, 102, 45, 52, 52, 97, 98, 45, 52, 102, 48, 52, 45, 98, 48, 100, 101, 45, 50, 50, 53, 99, 101, 55, 54, 50, 52, 51, 55, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370." peer_addr = [fd00:1122:3344:101::7]:64940 req_id = e713f987-aea9-48e3-b5e4-5a507141e369
892026-04-17T23:30:00.651ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56236, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), ), original: Query { name: Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 54, 102, 57, 50, 48, 100, 102, 45, 52, 52, 97, 98, 45, 52, 102, 48, 52, 45, 98, 48, 100, 101, 45, 50, 50, 53, 99, 101, 55, 54, 50, 52, 51, 55, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370." peer_addr = [fd00:1122:3344:101::7]:60751 req_id = a028d1a1-8130-493d-9f68-a13316ef45f2
902026-04-17T23:30:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44765, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a."), ), original: Query { name: Name("oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 51, 100, 102, 51, 54, 99, 51, 98, 45, 102, 53, 101, 100, 45, 52, 100, 54, 54, 45, 57, 99, 98, 97, 45, 56, 98, 98, 100, 51, 101, 55, 51, 55, 100, 56, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a." peer_addr = [fd00:1122:3344:101::1b]:46114 req_id = b85e680a-407c-48b1-8c60-e2a1d7d3d63f
912026-04-17T23:30:00.686ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), ), original: Query { name: Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 50, 51, 99, 98, 100, 98, 57, 45, 54, 101, 50, 50, 45, 52, 51, 98, 97, 45, 56, 101, 50, 50, 45, 53, 99, 53, 49, 52, 97, 97, 52, 55, 48, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049." peer_addr = [fd00:1122:3344:101::4]:40144 req_id = 3ca9e559-3fcc-44a3-90b0-1a49fb8c4d0f
922026-04-17T23:30:00.687ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), ), original: Query { name: Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 50, 51, 99, 98, 100, 98, 57, 45, 54, 101, 50, 50, 45, 52, 51, 98, 97, 45, 56, 101, 50, 50, 45, 53, 99, 53, 49, 52, 97, 97, 52, 55, 48, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049." peer_addr = [fd00:1122:3344:101::4]:39683 req_id = 56d3d3ae-f85b-4b0b-ae11-445e95e3bb9f