Max level shown:
1 [ Aug 19 01:27:26 Enabled. ]
2 [ Aug 19 01:27:26 Rereading configuration. ]
3 [ Aug 19 01:27:26 Rereading configuration. ]
4 [ Aug 19 01:27:28 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Aug 19 01:27:28 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-08-19T01:27:28.218ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-08-19T01:27:28.218ZINFOdns-server (store): opening sled database path = /data/dns
102026-08-19T01:27:28.242ZINFOdns-server (store): pruning trees for generations newer than 0
112026-08-19T01:27:28.242ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-08-19T01:27:28.242ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-08-19T01:27:28.242ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-08-19T01:27:30.139ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:37931
152026-08-19T01:27:30.140ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = ec768b6d-85aa-428a-a586-b63732f6d2f3
162026-08-19T01:27:30.140ZINFOdns-server (store): pruning trees for generations newer than 0
172026-08-19T01:27:30.161ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-08-19T01:27:30.161ZINFOdns-server (store): updated generation new_generation = 1 req_id = ec768b6d-85aa-428a-a586-b63732f6d2f3
192026-08-19T01:27:30.161ZINFOdns-server (http): request completed latency_us = 21338 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:37931 req_id = ec768b6d-85aa-428a-a586-b63732f6d2f3 response_code = 204 uri = /config
202026-08-19T01:27:41.342ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9568, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:35889 req_id = 4173d3aa-e1a0-45a1-a462-8a124ffb33b2
212026-08-19T01:27:41.346ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22198, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:43831 req_id = e46205d1-bb06-4a4b-93b7-7fede2e55acc
222026-08-19T01:27:41.351ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1918, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:61056 req_id = da13748d-a92b-4131-a5f1-8da7e0013a22
232026-08-19T01:29:29.495ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5810, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:44769 req_id = beee5ac5-9ce6-455b-8b34-14e50b40b379
242026-08-19T01:29:29.496ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54355, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:44828 req_id = 541e6876-3077-41cd-a44c-a21865e035ef
252026-08-19T01:29:29.497ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20763, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:38012 req_id = ae26bfac-9c9d-4466-a66e-5617f4c39aee
262026-08-19T01:29:34.547ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63459, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60235 req_id = 0afc5d77-8c7f-4b80-a268-b9c3b73e0681
272026-08-19T01:29:35.895ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45507, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), ), original: Query { name: Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 55, 55, 98, 54, 54, 102, 53, 57, 45, 99, 52, 98, 102, 45, 52, 101, 97, 56, 45, 97, 54, 97, 55, 45, 55, 57, 101, 56, 49, 97, 57, 51, 49, 100, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b." peer_addr = [fd00:1122:3344:101::e]:46930 req_id = 8b85efc2-0f15-4b71-ba5f-af77166be678
282026-08-19T01:29:35.895ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45507, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), ), original: Query { name: Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 55, 55, 98, 54, 54, 102, 53, 57, 45, 99, 52, 98, 102, 45, 52, 101, 97, 56, 45, 97, 54, 97, 55, 45, 55, 57, 101, 56, 49, 97, 57, 51, 49, 100, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b." peer_addr = [fd00:1122:3344:101::e]:50899 req_id = 764af951-0be1-4477-b713-f7969d2e5362
292026-08-19T01:29:44.548ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43634, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61217 req_id = ac6ed2f1-a825-4e38-b2d8-44012a689e98
302026-08-19T01:29:44.737ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2510, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:3718 req_id = b5e7a75c-ffac-4085-b14a-e5c2008de848
312026-08-19T01:29:44.886ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56196, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:13056 req_id = bee458f0-adf8-4b62-a3ce-7ec97be59822
322026-08-19T01:29:44.889ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65026, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:59138 req_id = 92472821-c0b0-42b7-982e-ee7124428cdd
332026-08-19T01:29:47.341ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20008, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:36341 req_id = e0f5348e-441c-4187-ba9c-3e9e52826fe4
342026-08-19T01:29:47.774ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:63447
352026-08-19T01:29:47.856ZINFOdns-server (http): request completed latency_us = 53893 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:63447 req_id = b3f81429-14ee-4498-a192-1335f4db4550 response_code = 200 uri = /config
362026-08-19T01:29:49.438ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:63736
372026-08-19T01:29:49.439ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 0eb1bc84-58c3-4e73-92ae-72451a952aac
382026-08-19T01:29:49.440ZINFOdns-server (store): updated generation new_generation = 1 req_id = 0eb1bc84-58c3-4e73-92ae-72451a952aac
392026-08-19T01:29:49.440ZINFOdns-server (http): request completed latency_us = 288 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:63736 req_id = 0eb1bc84-58c3-4e73-92ae-72451a952aac response_code = 204 uri = /config
402026-08-19T01:29:49.833ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6439, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:34072 req_id = 8fbe06b0-d800-4a66-a307-3ee27755bc12
412026-08-19T01:29:50.070ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:37624
422026-08-19T01:29:50.070ZINFOdns-server (http): request completed latency_us = 451 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:37624 req_id = 5341e9d0-e969-45d3-89ab-e22e163cb0a5 response_code = 200 uri = /config
432026-08-19T01:29:50.872ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22571, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:20497 req_id = 5957d1c6-dee0-4360-8063-330e037b4e00
442026-08-19T01:29:51.100ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:41792
452026-08-19T01:29:51.100ZINFOdns-server (http): request completed latency_us = 241 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:41792 req_id = a78a3824-2a76-47b4-9d96-4a53c87a9086 response_code = 200 uri = /config
462026-08-19T01:29:54.550ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22347, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52623 req_id = 90795b31-2815-4b13-88ab-16b9260311cb
472026-08-19T01:29:56.057ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12901, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:12190 req_id = ee500174-fc89-4a75-b556-67fb56718c95
482026-08-19T01:29:56.062ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22635, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:28925 req_id = cbc1a13b-89fd-4ecd-bf3b-971d77a4715a
492026-08-19T01:29:56.204ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:57714
502026-08-19T01:29:56.205ZINFOdns-server (http): request completed latency_us = 172 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:57714 req_id = 0e27b525-6a0f-48bd-9398-c82595c78eef response_code = 200 uri = /config
512026-08-19T01:29:56.273ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:41891
522026-08-19T01:29:56.274ZINFOdns-server (http): request completed latency_us = 192 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:41891 req_id = 7d8a5dd6-aa99-4b21-9e1f-f6a6a7273bab response_code = 200 uri = /config
532026-08-19T01:30:00.143ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47057, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_14582a57-0a33-4667-a7bf-1c75475f1b5d."), ), original: Query { name: Name("oxz_cockroachdb_14582a57-0a33-4667-a7bf-1c75475f1b5d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 52, 53, 56, 50, 97, 53, 55, 45, 48, 97, 51, 51, 45, 52, 54, 54, 55, 45, 97, 55, 98, 102, 45, 49, 99, 55, 53, 52, 55, 53, 102, 49, 98, 53, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_14582a57-0a33-4667-a7bf-1c75475f1b5d." peer_addr = [fd00:1122:3344:101::7]:43779 req_id = 82330196-4c2c-4fb0-baea-9626124461bb
542026-08-19T01:30:00.143ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47057, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_14582a57-0a33-4667-a7bf-1c75475f1b5d."), ), original: Query { name: Name("oxz_cockroachdb_14582a57-0a33-4667-a7bf-1c75475f1b5d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 52, 53, 56, 50, 97, 53, 55, 45, 48, 97, 51, 51, 45, 52, 54, 54, 55, 45, 97, 55, 98, 102, 45, 49, 99, 55, 53, 52, 55, 53, 102, 49, 98, 53, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_14582a57-0a33-4667-a7bf-1c75475f1b5d." peer_addr = [fd00:1122:3344:101::7]:54919 req_id = 7d37f1b1-f4e8-4f23-a191-a98fd8b9a5a0
552026-08-19T01:30:00.177ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63750, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8f339db8-5169-4262-b68e-6ce1a3f87d94."), ), original: Query { name: Name("oxz_cockroachdb_8f339db8-5169-4262-b68e-6ce1a3f87d94."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 102, 51, 51, 57, 100, 98, 56, 45, 53, 49, 54, 57, 45, 52, 50, 54, 50, 45, 98, 54, 56, 101, 45, 54, 99, 101, 49, 97, 51, 102, 56, 55, 100, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8f339db8-5169-4262-b68e-6ce1a3f87d94." peer_addr = [fd00:1122:3344:101::6]:37031 req_id = 1e2f813e-0680-41c3-b306-261a0ef10b4a
562026-08-19T01:30:00.178ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63750, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_8f339db8-5169-4262-b68e-6ce1a3f87d94."), ), original: Query { name: Name("oxz_cockroachdb_8f339db8-5169-4262-b68e-6ce1a3f87d94."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 102, 51, 51, 57, 100, 98, 56, 45, 53, 49, 54, 57, 45, 52, 50, 54, 50, 45, 98, 54, 56, 101, 45, 54, 99, 101, 49, 97, 51, 102, 56, 55, 100, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_8f339db8-5169-4262-b68e-6ce1a3f87d94." peer_addr = [fd00:1122:3344:101::6]:47918 req_id = a09e6f00-6d89-4c8b-ba12-faa74be16df1
572026-08-19T01:30:00.352ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39721, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:45918 req_id = 115c0452-6cfe-4bde-b045-9a924c86911d
582026-08-19T01:30:00.353ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39721, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:41028 req_id = a6da4da7-4e00-4f07-b978-52dac20b617e
592026-08-19T01:30:00.366ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44168, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_87f38c7b-4250-4549-9b3c-b7c2e0e19503."), ), original: Query { name: Name("oxz_ntp_87f38c7b-4250-4549-9b3c-b7c2e0e19503."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 56, 55, 102, 51, 56, 99, 55, 98, 45, 52, 50, 53, 48, 45, 52, 53, 52, 57, 45, 57, 98, 51, 99, 45, 98, 55, 99, 50, 101, 48, 101, 49, 57, 53, 48, 51, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_87f38c7b-4250-4549-9b3c-b7c2e0e19503." peer_addr = [fd00:1122:3344:101::1b]:37667 req_id = 48cbb5df-ea13-47c7-810a-91341e16057a
602026-08-19T01:30:00.566ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35957, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), ), original: Query { name: Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 55, 55, 98, 54, 54, 102, 53, 57, 45, 99, 52, 98, 102, 45, 52, 101, 97, 56, 45, 97, 54, 97, 55, 45, 55, 57, 101, 56, 49, 97, 57, 51, 49, 100, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b." peer_addr = [fd00:1122:3344:101::e]:33087 req_id = e607b2d9-aa18-4e92-b6c4-097c4c8ae076
612026-08-19T01:30:00.567ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35957, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), ), original: Query { name: Name("oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 55, 55, 98, 54, 54, 102, 53, 57, 45, 99, 52, 98, 102, 45, 52, 101, 97, 56, 45, 97, 54, 97, 55, 45, 55, 57, 101, 56, 49, 97, 57, 51, 49, 100, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_77b66f59-c4bf-4ea8-a6a7-79e81a931d7b." peer_addr = [fd00:1122:3344:101::e]:52721 req_id = 15ba2ede-893e-4bcc-93a0-8e685786e131