Max level shown:
1 [ Jul 11 17:56:35 Enabled. ]
2 [ Jul 11 17:56:35 Rereading configuration. ]
3 [ Jul 11 17:56:36 Rereading configuration. ]
4 [ Jul 11 17:56:38 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Jul 11 17:56:38 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-11T17:56:38.323ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-07-11T17:56:38.323ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-11T17:56:38.341ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-11T17:56:38.341ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-11T17:56:38.341ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-07-11T17:56:38.342ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-07-11T17:56:38.422ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:44810
152026-07-11T17:56:38.422ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 1d8fc0ea-2143-4202-bd1f-fb32ee7d37c6
162026-07-11T17:56:38.422ZINFOdns-server (store): pruning trees for generations newer than 0
172026-07-11T17:56:38.472ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-07-11T17:56:38.472ZINFOdns-server (store): updated generation new_generation = 1 req_id = 1d8fc0ea-2143-4202-bd1f-fb32ee7d37c6
192026-07-11T17:56:38.472ZINFOdns-server (http): request completed latency_us = 49972 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:44810 req_id = 1d8fc0ea-2143-4202-bd1f-fb32ee7d37c6 response_code = 204 uri = /config
202026-07-11T17:56:51.696ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 770, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:54190 req_id = 80f9bf61-6d2b-435f-81da-9939326a6901
212026-07-11T17:56:51.701ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14116, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:60370 req_id = 1b287e5e-e826-4b64-9012-1d6984236335
222026-07-11T17:56:51.705ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23629, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:61748 req_id = 830a3edb-0198-43fa-a5ee-16c3aedaef61
232026-07-11T17:58:23.417ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21651, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62295 req_id = 7b7e62cc-3169-4c16-b02f-8fb9712a5301
242026-07-11T17:58:26.779ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63857, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:55690 req_id = 1a0f27fd-67c1-457a-bf88-4305c8a94dfa
252026-07-11T17:58:26.784ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63857, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:50496 req_id = c499134c-8ced-4554-a9d2-a9e1cb43824f
262026-07-11T17:58:31.568ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64281, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:17483 req_id = 73cb2758-6a9f-4f4a-986a-a00e328c88e5
272026-07-11T17:58:32.801ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8508, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:50531 req_id = 5a84e5af-de88-4418-b41b-37f65067e0ed
282026-07-11T17:58:32.825ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2058, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:31278 req_id = d509aec0-7427-43ec-bb44-0802294cab8d
292026-07-11T17:58:33.400ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30289, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64034 req_id = 54371b6f-2f51-452d-a7c8-1c294efa56de
302026-07-11T17:58:43.402ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44397, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58667 req_id = ce44f40f-7234-4896-894a-6235d0c3194e
312026-07-11T17:58:53.404ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41162, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56322 req_id = cdaa2932-d45b-4174-99cb-057908f986c0
322026-07-11T17:58:55.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41751, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:27736 req_id = 890eb1ce-683d-4ec5-83dc-30c7ef4a8534
332026-07-11T17:58:55.952ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:46383
342026-07-11T17:58:56.021ZINFOdns-server (http): request completed latency_us = 45490 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:46383 req_id = 80b659cb-58a2-430a-b856-7b4f59eb2535 response_code = 200 uri = /config
352026-07-11T17:59:00.465ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38892, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:31932 req_id = ecee0251-824c-4006-b8e8-2a86d4fc4edc
362026-07-11T17:59:00.485ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:44527
372026-07-11T17:59:00.486ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = d0d7bca3-c64e-497c-a68c-5313933c6ac7
382026-07-11T17:59:00.486ZINFOdns-server (store): updated generation new_generation = 1 req_id = d0d7bca3-c64e-497c-a68c-5313933c6ac7
392026-07-11T17:59:00.486ZINFOdns-server (http): request completed latency_us = 343 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:44527 req_id = d0d7bca3-c64e-497c-a68c-5313933c6ac7 response_code = 204 uri = /config
402026-07-11T17:59:00.761ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:42453
412026-07-11T17:59:00.761ZINFOdns-server (http): request completed latency_us = 243 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:42453 req_id = d712ea0c-cf57-4319-907f-429487d42837 response_code = 200 uri = /config
422026-07-11T17:59:01.661ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49018, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:6332 req_id = afe678dd-f632-417f-896b-578506626e3d
432026-07-11T17:59:01.905ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:62174
442026-07-11T17:59:01.905ZINFOdns-server (http): request completed latency_us = 242 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:62174 req_id = 3efa6cae-41f3-4279-b492-cc32f9fc8f25 response_code = 200 uri = /config
452026-07-11T17:59:02.387ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32758, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:12273 req_id = 706d1ef4-5979-4759-8a5c-9580e2c193c8
462026-07-11T17:59:02.520ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:61543
472026-07-11T17:59:02.521ZINFOdns-server (http): request completed latency_us = 223 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:61543 req_id = bde18241-6e41-4036-ad3f-55107fbbca6a response_code = 200 uri = /config
482026-07-11T17:59:03.406ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38131, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50892 req_id = ee38912f-d9a2-40a8-a879-27c71e6f0a2d
492026-07-11T17:59:03.752ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36859, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:57237 req_id = 16e657ce-78d1-4fe9-b445-931d8fbb814c
502026-07-11T17:59:03.881ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:54548
512026-07-11T17:59:03.881ZINFOdns-server (http): request completed latency_us = 191 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:54548 req_id = ff4ac825-771f-4a26-88d6-3fe66797012b response_code = 200 uri = /config
522026-07-11T17:59:13.408ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10302, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55647 req_id = a2496276-98cd-4e88-a03c-a60e21fa2f57
532026-07-11T17:59:23.409ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61593, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58987 req_id = a751f67e-1705-4475-8ec0-04a735ecb14e
542026-07-11T17:59:31.484ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:51695
552026-07-11T17:59:31.486ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 4a3a2413-fb28-4292-9d9a-548f3b5713f8
562026-07-11T17:59:31.486ZINFOdns-server (store): updated generation new_generation = 1 req_id = 4a3a2413-fb28-4292-9d9a-548f3b5713f8
572026-07-11T17:59:31.486ZINFOdns-server (http): request completed latency_us = 956 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:51695 req_id = 4a3a2413-fb28-4292-9d9a-548f3b5713f8 response_code = 204 uri = /config
582026-07-11T17:59:32.521ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:65150
592026-07-11T17:59:32.523ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 5ef06ab9-8eab-4e3a-b79c-033fd369e164
602026-07-11T17:59:32.523ZINFOdns-server (store): updated generation new_generation = 1 req_id = 5ef06ab9-8eab-4e3a-b79c-033fd369e164
612026-07-11T17:59:32.523ZINFOdns-server (http): request completed latency_us = 249 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:65150 req_id = 5ef06ab9-8eab-4e3a-b79c-033fd369e164 response_code = 204 uri = /config
622026-07-11T17:59:32.827ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:34927
632026-07-11T17:59:32.829ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = ff799ee0-077e-4f04-a291-00ad020a7a2c
642026-07-11T17:59:32.829ZINFOdns-server (store): updated generation new_generation = 1 req_id = ff799ee0-077e-4f04-a291-00ad020a7a2c
652026-07-11T17:59:32.829ZINFOdns-server (http): request completed latency_us = 296 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:34927 req_id = ff799ee0-077e-4f04-a291-00ad020a7a2c response_code = 204 uri = /config
662026-07-11T17:59:33.193ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62879, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:56999 req_id = aea735da-154d-48e2-9e60-ecf7d0d81ba5
672026-07-11T17:59:33.297ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:55067
682026-07-11T17:59:33.297ZINFOdns-server (http): request completed latency_us = 190 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:55067 req_id = b0b93b3f-b7d0-45be-943d-688a1f67dddd response_code = 200 uri = /config
692026-07-11T17:59:33.411ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47696, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54601 req_id = 04d3bbce-55cd-4620-9ee9-a7d42fd25b23
702026-07-11T17:59:33.778ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36959, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:49771 req_id = 6d276d00-0647-41cd-b86d-c0046292ba38
712026-07-11T17:59:33.839ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1221, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:42561 req_id = 96d782ea-cbea-4a94-b708-4bf7efc3192d
722026-07-11T17:59:33.944ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:41847
732026-07-11T17:59:33.945ZINFOdns-server (http): request completed latency_us = 219 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:41847 req_id = 908ca432-0731-444b-bd74-5f66244b7ff9 response_code = 200 uri = /config
742026-07-11T17:59:33.986ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:56025
752026-07-11T17:59:33.986ZINFOdns-server (http): request completed latency_us = 208 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:56025 req_id = e0dd6952-79d0-42be-8501-4cdf05098e3d response_code = 200 uri = /config
762026-07-11T17:59:43.413ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33377, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53041 req_id = c961c91c-6e5d-4371-91e5-f12c73eb65f9
772026-07-11T17:59:53.414ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14690, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63677 req_id = a029bdde-81d4-4b72-a812-a1c0fc354753
782026-07-11T18:00:00.099ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:57046 req_id = f1475206-6810-4991-be42-e534ae2c7787
792026-07-11T18:00:00.100ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:62928 req_id = 3660ee02-9c51-48b5-ae18-7a71229fd543
802026-07-11T18:00:00.486ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14642, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_cf56f114-d8ea-4de5-9868-dbbf52c8d8e0."), ), original: Query { name: Name("oxz_ntp_cf56f114-d8ea-4de5-9868-dbbf52c8d8e0."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 99, 102, 53, 54, 102, 49, 49, 52, 45, 100, 56, 101, 97, 45, 52, 100, 101, 53, 45, 57, 56, 54, 56, 45, 100, 98, 98, 102, 53, 50, 99, 56, 100, 56, 101, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_cf56f114-d8ea-4de5-9868-dbbf52c8d8e0." peer_addr = [fd00:1122:3344:101::1b]:49431 req_id = 3cd8ee57-6cca-478e-a9c0-8d5cd097a1c6
812026-07-11T18:00:00.661ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60549, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:53750 req_id = f7a85d9a-e267-4722-90ad-b89cd1813067
822026-07-11T18:00:00.661ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60549, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:56706 req_id = c7aa8bdd-fd55-4335-b5dd-217f54782abf
832026-07-11T18:00:00.805ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19565, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), ), original: Query { name: Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 57, 99, 56, 50, 54, 53, 55, 45, 101, 53, 101, 51, 45, 52, 100, 57, 100, 45, 56, 51, 99, 51, 45, 102, 55, 52, 101, 48, 98, 50, 52, 48, 102, 50, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24." peer_addr = [fd00:1122:3344:101::6]:55809 req_id = 1dd2a087-a65e-4f02-9afc-7220e472d8eb
842026-07-11T18:00:00.806ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19565, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), ), original: Query { name: Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 57, 99, 56, 50, 54, 53, 55, 45, 101, 53, 101, 51, 45, 52, 100, 57, 100, 45, 56, 51, 99, 51, 45, 102, 55, 52, 101, 48, 98, 50, 52, 48, 102, 50, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24." peer_addr = [fd00:1122:3344:101::6]:45488 req_id = bc6a8db8-6e9d-41af-9ae7-28fa490ea432
852026-07-11T18:00:00.832ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65052, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), ), original: Query { name: Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 52, 50, 49, 55, 97, 50, 98, 45, 54, 97, 99, 55, 45, 52, 98, 100, 100, 45, 97, 97, 55, 56, 45, 101, 53, 49, 52, 54, 100, 56, 51, 99, 57, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949." peer_addr = [fd00:1122:3344:101::5]:64552 req_id = 2b8259d4-b2e3-4cc9-b220-4949cc91c643
862026-07-11T18:00:00.833ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65052, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), ), original: Query { name: Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 52, 50, 49, 55, 97, 50, 98, 45, 54, 97, 99, 55, 45, 52, 98, 100, 100, 45, 97, 97, 55, 56, 45, 101, 53, 49, 52, 54, 100, 56, 51, 99, 57, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949." peer_addr = [fd00:1122:3344:101::5]:34513 req_id = f800cc70-488a-442a-95fd-5e0ddfbb0a54
872026-07-11T18:00:00.891ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8485, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), ), original: Query { name: Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 100, 100, 52, 49, 57, 101, 49, 45, 97, 98, 100, 57, 45, 52, 52, 49, 52, 45, 56, 50, 51, 101, 45, 49, 54, 50, 56, 102, 56, 98, 98, 51, 49, 54, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167." peer_addr = [fd00:1122:3344:101::7]:63153 req_id = b06a9bc7-78c6-4936-8370-802fab24383b
882026-07-11T18:00:00.891ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8485, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), ), original: Query { name: Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 100, 100, 52, 49, 57, 101, 49, 45, 97, 98, 100, 57, 45, 52, 52, 49, 52, 45, 56, 50, 51, 101, 45, 49, 54, 50, 56, 102, 56, 98, 98, 51, 49, 54, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167." peer_addr = [fd00:1122:3344:101::7]:44705 req_id = 83b4052a-d8b7-483a-9e88-d375e2b6418a
892026-07-11T18:00:00.965ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), ), original: Query { name: Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 54, 56, 50, 54, 53, 98, 51, 45, 97, 50, 100, 99, 45, 52, 98, 54, 102, 45, 97, 49, 99, 101, 45, 54, 55, 97, 102, 100, 54, 57, 50, 97, 49, 97, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5." peer_addr = [fd00:1122:3344:101::3]:41264 req_id = 666a7c6d-9ca6-4efa-bff6-59cc38567aad
902026-07-11T18:00:00.965ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), ), original: Query { name: Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 54, 56, 50, 54, 53, 98, 51, 45, 97, 50, 100, 99, 45, 52, 98, 54, 102, 45, 97, 49, 99, 101, 45, 54, 55, 97, 102, 100, 54, 57, 50, 97, 49, 97, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5." peer_addr = [fd00:1122:3344:101::3]:39883 req_id = b1439dd4-e19c-42ee-afc0-7bddb33c4263
912026-07-11T18:00:00.968ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22323, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), ), original: Query { name: Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 57, 57, 50, 97, 101, 53, 54, 45, 99, 53, 54, 98, 45, 52, 98, 102, 55, 45, 56, 101, 56, 49, 45, 100, 101, 98, 52, 98, 97, 101, 99, 56, 98, 51, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d." peer_addr = [fd00:1122:3344:101::4]:63683 req_id = bae30f0b-37c5-4018-8cba-5f694cc7f0a1
922026-07-11T18:00:00.969ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22323, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), ), original: Query { name: Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 57, 57, 50, 97, 101, 53, 54, 45, 99, 53, 54, 98, 45, 52, 98, 102, 55, 45, 56, 101, 56, 49, 45, 100, 101, 98, 52, 98, 97, 101, 99, 56, 98, 51, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d." peer_addr = [fd00:1122:3344:101::4]:59876 req_id = 74e63939-604d-426b-9612-220a9500b5a6