Max level shown:
1 [ Apr 17 23:26:23 Enabled. ]
2 [ Apr 17 23:26:23 Rereading configuration. ]
3 [ Apr 17 23:26:24 Rereading configuration. ]
4 [ Apr 17 23:26:25 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Apr 17 23:26:25 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-04-17T23:26:25.469ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-04-17T23:26:25.469ZINFOdns-server (store): opening sled database path = /data/dns
102026-04-17T23:26:25.488ZINFOdns-server (store): pruning trees for generations newer than 0
112026-04-17T23:26:25.488ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-04-17T23:26:25.488ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-04-17T23:26:25.489ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-04-17T23:26:26.292ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:56543
152026-04-17T23:26:26.292ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = a7f5a22e-5b56-490f-8d34-b0accad79175
162026-04-17T23:26:26.292ZINFOdns-server (store): pruning trees for generations newer than 0
172026-04-17T23:26:26.308ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-04-17T23:26:26.308ZINFOdns-server (store): updated generation new_generation = 1 req_id = a7f5a22e-5b56-490f-8d34-b0accad79175
192026-04-17T23:26:26.308ZINFOdns-server (http): request completed latency_us = 16330 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:56543 req_id = a7f5a22e-5b56-490f-8d34-b0accad79175 response_code = 204 uri = /config
202026-04-17T23:26:38.755ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32439, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:55063 req_id = c1110db2-c0e6-48ce-bb13-d9450a923bc3
212026-04-17T23:26:38.760ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18032, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:52903 req_id = df5d0cc8-ecd2-4945-a3f8-a7b1d10b4ade
222026-04-17T23:26:38.764ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4463, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:63703 req_id = b8b18010-94a8-43cd-bc0e-2891dd49eb10
232026-04-17T23:28:28.018ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14940, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51130 req_id = 1f88113f-d948-4953-9b3c-afc57004a672
242026-04-17T23:28:31.872ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:56642 req_id = 5307dece-b8aa-4977-ad57-33b30fdbba3f
252026-04-17T23:28:31.873ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16873, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:61448 req_id = b16e0eba-dcba-472d-85f3-e5490cf5938d
262026-04-17T23:28:38.002ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44387, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52542 req_id = c350a6c9-f5bd-460d-b82c-13fb61fefe1f
272026-04-17T23:28:38.195ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21253, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:18723 req_id = 2bf91508-9df3-4dc8-a1e6-1107ba31e587
282026-04-17T23:28:38.197ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29719, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:54462 req_id = 1fa2ac91-c2c1-439b-85d9-ac40e8f38f44
292026-04-17T23:28:38.198ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59257, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:2073 req_id = 80554c8f-f99f-4c7a-8311-6f14fc01c961
302026-04-17T23:28:45.826ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14566, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:56105 req_id = 2a82bf3a-0444-4df1-9b62-c5453c7a0721
312026-04-17T23:28:46.120ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:35566
322026-04-17T23:28:46.169ZINFOdns-server (http): request completed latency_us = 26436 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:35566 req_id = 02ea372b-7680-46f4-b2fc-d46b99eb4339 response_code = 200 uri = /config
332026-04-17T23:28:48.005ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12454, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56030 req_id = 8766a8f6-c24a-49fc-8d5f-a74a9f3ad701
342026-04-17T23:28:48.168ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:43574
352026-04-17T23:28:48.175ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 0b5915fa-3b0b-4485-a6bc-8f8fbaea10b8
362026-04-17T23:28:48.175ZINFOdns-server (store): updated generation new_generation = 1 req_id = 0b5915fa-3b0b-4485-a6bc-8f8fbaea10b8
372026-04-17T23:28:48.175ZINFOdns-server (http): request completed latency_us = 5349 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:43574 req_id = 0b5915fa-3b0b-4485-a6bc-8f8fbaea10b8 response_code = 204 uri = /config
382026-04-17T23:28:48.198ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57330, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:3402 req_id = 87a71fe3-7db2-46e4-b96a-9b964255c8db
392026-04-17T23:28:48.440ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:49613
402026-04-17T23:28:48.440ZINFOdns-server (http): request completed latency_us = 219 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:49613 req_id = fafab549-ff20-43d4-8bed-a462b04cb69b response_code = 200 uri = /config
412026-04-17T23:28:49.197ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57074, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:34588 req_id = 21c99d28-e9a8-4d3f-91df-0138ae6007ab
422026-04-17T23:28:49.198ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58496, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:36429 req_id = 88d9baae-3772-46e7-a19e-b4cefd2c2a27
432026-04-17T23:28:49.347ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:42151
442026-04-17T23:28:49.347ZINFOdns-server (http): request completed latency_us = 267 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:42151 req_id = 66d0d50b-4e30-4580-9d56-2cfcbf8e2e28 response_code = 200 uri = /config
452026-04-17T23:28:49.446ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:40693
462026-04-17T23:28:49.447ZINFOdns-server (http): request completed latency_us = 181 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:40693 req_id = 491b5fa6-db10-4165-90fb-eecf0940d9da response_code = 200 uri = /config
472026-04-17T23:28:50.785ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22811, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:4566 req_id = 436a23f0-69c3-4807-9e55-50b5cadd993e
482026-04-17T23:28:51.000ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:51193
492026-04-17T23:28:51.000ZINFOdns-server (http): request completed latency_us = 215 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:51193 req_id = bbe0ded0-553c-4c34-a69a-37bfb94b8a87 response_code = 200 uri = /config
502026-04-17T23:28:58.007ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9166, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64125 req_id = 56a66c56-6d90-4231-884e-77e676ff292f
512026-04-17T23:29:08.010ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10124, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63255 req_id = 1d492579-60d5-4f76-9c8c-31fe9e04a366
522026-04-17T23:29:18.011ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59908, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50888 req_id = a33181ed-6cd7-4160-b4f1-9681466b016e
532026-04-17T23:29:28.012ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3623, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61803 req_id = e2cffedf-8573-4218-8c4b-dd0bd9e6b220
542026-04-17T23:29:37.859ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:38302
552026-04-17T23:29:37.861ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 50f26e68-46fa-450f-ad84-b598f43b996d
562026-04-17T23:29:37.861ZINFOdns-server (store): updated generation new_generation = 1 req_id = 50f26e68-46fa-450f-ad84-b598f43b996d
572026-04-17T23:29:37.861ZINFOdns-server (http): request completed latency_us = 355 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:38302 req_id = 50f26e68-46fa-450f-ad84-b598f43b996d response_code = 204 uri = /config
582026-04-17T23:29:38.014ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31807, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:65338 req_id = 1264d35e-d9ea-46d5-8a8b-4b526e3eb41e
592026-04-17T23:29:38.084ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:45428
602026-04-17T23:29:38.085ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = d891ed68-1332-4ec5-a0cd-78bf7d0a4798
612026-04-17T23:29:38.085ZINFOdns-server (store): updated generation new_generation = 1 req_id = d891ed68-1332-4ec5-a0cd-78bf7d0a4798
622026-04-17T23:29:38.085ZINFOdns-server (http): request completed latency_us = 269 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:45428 req_id = d891ed68-1332-4ec5-a0cd-78bf7d0a4798 response_code = 204 uri = /config
632026-04-17T23:29:38.276ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:54924
642026-04-17T23:29:38.278ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 85219e76-4d48-4524-bdfb-055810ae1a04
652026-04-17T23:29:38.278ZINFOdns-server (store): updated generation new_generation = 1 req_id = 85219e76-4d48-4524-bdfb-055810ae1a04
662026-04-17T23:29:38.278ZINFOdns-server (http): request completed latency_us = 351 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:54924 req_id = 85219e76-4d48-4524-bdfb-055810ae1a04 response_code = 204 uri = /config
672026-04-17T23:29:39.249ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63476, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:58468 req_id = 25427236-812e-44b4-8968-6d47ebc8147e
682026-04-17T23:29:39.309ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38962, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:22958 req_id = db07a8db-6412-4164-a83b-9bd2f5f08cb4
692026-04-17T23:29:39.349ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:54345
702026-04-17T23:29:39.350ZINFOdns-server (http): request completed latency_us = 262 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:54345 req_id = 7ee23e74-d49a-40e6-aa7f-27a73a0de691 response_code = 200 uri = /config
712026-04-17T23:29:39.494ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:35907
722026-04-17T23:29:39.494ZINFOdns-server (http): request completed latency_us = 242 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:35907 req_id = 7108ea60-2b3d-48e8-9c4b-a5337e74727d response_code = 200 uri = /config
732026-04-17T23:29:40.049ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9824, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:38528 req_id = 8f738d5d-7671-4a60-8715-b3316767dfe8
742026-04-17T23:29:40.234ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:43113
752026-04-17T23:29:40.234ZINFOdns-server (http): request completed latency_us = 192 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:43113 req_id = fce0aafc-7353-4da4-8e08-2860c3cefc9e response_code = 200 uri = /config
762026-04-17T23:29:48.015ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2683, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59055 req_id = 4fa934d1-b9da-4257-8c7d-3962dad6e382
772026-04-17T23:29:58.018ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56647, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53824 req_id = a46c3e6c-9f90-4900-9a02-47e7574c2c01
782026-04-17T23:30:00.206ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36603, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:36496 req_id = b7f78a29-7624-4cdb-8622-cb535ed6b632
792026-04-17T23:30:00.207ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36603, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), ), original: Query { name: Name("oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 101, 55, 99, 50, 48, 50, 97, 101, 45, 52, 99, 51, 50, 45, 52, 54, 49, 52, 45, 57, 57, 99, 52, 45, 49, 50, 55, 101, 54, 99, 102, 54, 101, 98, 101, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_e7c202ae-4c32-4614-99c4-127e6cf6ebee." peer_addr = [fd00:1122:3344:101::e]:51621 req_id = 2b0caa97-8c5e-4558-8ff6-932ef7723357
802026-04-17T23:30:00.564ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), ), original: Query { name: Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 101, 98, 102, 99, 51, 52, 53, 45, 50, 55, 51, 49, 45, 52, 100, 100, 50, 45, 97, 98, 99, 53, 45, 51, 57, 55, 98, 98, 55, 52, 100, 53, 54, 50, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b." peer_addr = [fd00:1122:3344:101::3]:53879 req_id = b5c07e4c-8908-4f64-8959-e72bde44ffe8
812026-04-17T23:30:00.565ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7107, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), ), original: Query { name: Name("oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 101, 98, 102, 99, 51, 52, 53, 45, 50, 55, 51, 49, 45, 52, 100, 100, 50, 45, 97, 98, 99, 53, 45, 51, 57, 55, 98, 98, 55, 52, 100, 53, 54, 50, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8ebfc345-2731-4dd2-abc5-397bb74d562b." peer_addr = [fd00:1122:3344:101::3]:52160 req_id = f7434d6e-2c6d-45c3-a7f0-ffdec9dae203
822026-04-17T23:30:00.569ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), ), original: Query { name: Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 50, 98, 53, 57, 54, 101, 56, 45, 100, 49, 99, 56, 45, 52, 57, 100, 57, 45, 57, 52, 48, 102, 45, 49, 48, 98, 52, 50, 50, 101, 53, 102, 101, 56, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80." peer_addr = [fd00:1122:3344:101::5]:47649 req_id = 6748f7bf-1b01-49b5-afe4-d7a69f0c6268
832026-04-17T23:30:00.570ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), ), original: Query { name: Name("oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 50, 98, 53, 57, 54, 101, 56, 45, 100, 49, 99, 56, 45, 52, 57, 100, 57, 45, 57, 52, 48, 102, 45, 49, 48, 98, 52, 50, 50, 101, 53, 102, 101, 56, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_52b596e8-d1c8-49d9-940f-10b422e5fe80." peer_addr = [fd00:1122:3344:101::5]:61614 req_id = d4bc5cf7-c3bc-4eeb-92d6-2485945ce874
842026-04-17T23:30:00.596ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43303, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:62951 req_id = 9f112cf0-5b7b-47b4-8b9f-dd69431b4fb7
852026-04-17T23:30:00.597ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43303, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:51315 req_id = 7eeaa712-a5b9-4e1e-a138-07b22aec1f9c
862026-04-17T23:30:00.630ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22641, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), ), original: Query { name: Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 57, 55, 49, 101, 48, 51, 55, 45, 56, 57, 99, 99, 45, 52, 50, 99, 52, 45, 56, 53, 101, 102, 45, 102, 48, 97, 50, 54, 97, 56, 54, 51, 57, 53, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954." peer_addr = [fd00:1122:3344:101::6]:57951 req_id = f179df65-dd7a-432c-864c-edfb85b68bfd
872026-04-17T23:30:00.631ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22641, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), ), original: Query { name: Name("oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 57, 55, 49, 101, 48, 51, 55, 45, 56, 57, 99, 99, 45, 52, 50, 99, 52, 45, 56, 53, 101, 102, 45, 102, 48, 97, 50, 54, 97, 56, 54, 51, 57, 53, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_6971e037-89cc-42c4-85ef-f0a26a863954." peer_addr = [fd00:1122:3344:101::6]:33813 req_id = f1751865-0e7d-4d3f-bbf3-1f9fa44952a7
882026-04-17T23:30:00.651ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56236, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), ), original: Query { name: Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 54, 102, 57, 50, 48, 100, 102, 45, 52, 52, 97, 98, 45, 52, 102, 48, 52, 45, 98, 48, 100, 101, 45, 50, 50, 53, 99, 101, 55, 54, 50, 52, 51, 55, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370." peer_addr = [fd00:1122:3344:101::7]:36306 req_id = 2bc19e6e-0e21-4bb4-a28d-511523b77495
892026-04-17T23:30:00.652ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56236, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), ), original: Query { name: Name("oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 54, 102, 57, 50, 48, 100, 102, 45, 52, 52, 97, 98, 45, 52, 102, 48, 52, 45, 98, 48, 100, 101, 45, 50, 50, 53, 99, 101, 55, 54, 50, 52, 51, 55, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e6f920df-44ab-4f04-b0de-225ce7624370." peer_addr = [fd00:1122:3344:101::7]:55885 req_id = cd151b6a-5d5c-4717-bba6-3fc0fbc246a0
902026-04-17T23:30:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44765, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a."), ), original: Query { name: Name("oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 51, 100, 102, 51, 54, 99, 51, 98, 45, 102, 53, 101, 100, 45, 52, 100, 54, 54, 45, 57, 99, 98, 97, 45, 56, 98, 98, 100, 51, 101, 55, 51, 55, 100, 56, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_3df36c3b-f5ed-4d66-9cba-8bbd3e737d8a." peer_addr = [fd00:1122:3344:101::1b]:57272 req_id = 68912cbf-5074-43ed-9d68-111ee468f661
912026-04-17T23:30:00.686ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), ), original: Query { name: Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 50, 51, 99, 98, 100, 98, 57, 45, 54, 101, 50, 50, 45, 52, 51, 98, 97, 45, 56, 101, 50, 50, 45, 53, 99, 53, 49, 52, 97, 97, 52, 55, 48, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049." peer_addr = [fd00:1122:3344:101::4]:52848 req_id = cda3a2a4-d2b2-4313-b62f-dd36b8ed4cc1
922026-04-17T23:30:00.687ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), ), original: Query { name: Name("oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 50, 51, 99, 98, 100, 98, 57, 45, 54, 101, 50, 50, 45, 52, 51, 98, 97, 45, 56, 101, 50, 50, 45, 53, 99, 53, 49, 52, 97, 97, 52, 55, 48, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e23cbdb9-6e22-43ba-8e22-5c514aa47049." peer_addr = [fd00:1122:3344:101::4]:37615 req_id = 13ecdb63-2670-4117-b082-ad277b66e3e7