Max level shown:
1 [ Jul 2 03:42:52 Enabled. ]
2 [ Jul 2 03:42:52 Rereading configuration. ]
3 [ Jul 2 03:42:53 Rereading configuration. ]
4 [ Jul 2 03:42:54 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:1::1]:5353 --dns-address [fd00:1122:3344:1::1]:53 &"). ]
5 [ Jul 2 03:42:54 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-02T03:42:54.525ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:1::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:1::1]:53 }
92026-07-02T03:42:54.525ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-02T03:42:54.547ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-02T03:42:54.547ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-02T03:42:54.547ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:1::1]:53
132026-07-02T03:42:54.547ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:1::1]:5353
142026-07-02T03:42:54.671ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60058, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:7929 req_id = 8e643f63-533b-4132-9f53-41a7f1af1b53
152026-07-02T03:42:54.846ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63491, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:3982 req_id = 6689bfbd-a3b5-4277-8afa-7811cc4b793c
162026-07-02T03:42:55.076ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:1::1]:5353 remote_addr = [fd00:1122:3344:1::2]:50490
172026-07-02T03:42:55.076ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 0ab7c742-a5a1-4cdd-b1b5-8826d2c65a54
182026-07-02T03:42:55.077ZINFOdns-server (store): pruning trees for generations newer than 0
192026-07-02T03:42:55.089ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
202026-07-02T03:42:55.090ZINFOdns-server (store): updated generation new_generation = 1 req_id = 0ab7c742-a5a1-4cdd-b1b5-8826d2c65a54
212026-07-02T03:42:55.090ZINFOdns-server (http): request completed latency_us = 13352 local_addr = [fd00:1122:3344:1::1]:5353 method = PUT remote_addr = [fd00:1122:3344:1::2]:50490 req_id = 0ab7c742-a5a1-4cdd-b1b5-8826d2c65a54 response_code = 204 uri = /config
222026-07-02T03:43:07.923ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13904, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:33612 req_id = 1828018c-d123-4917-8977-6ce6e8e2cbca
232026-07-02T03:43:07.928ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45084, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:40266 req_id = 63ed7d80-e24b-484d-b85a-34bd7f3741cb
242026-07-02T03:43:07.933ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39414, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:65426 req_id = e2596a8c-5069-416f-9156-441197c0e731
252026-07-02T03:44:19.326ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52634, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64498 req_id = 8b96d0f7-af4d-438d-a77d-516c6943bd90
262026-07-02T03:44:20.958ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:39056 req_id = b7b8be7d-72df-4919-9265-2db6bd4d8ea0
272026-07-02T03:44:20.964ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:51702 req_id = ce7d55a2-d714-48d7-aee0-53d145d9ed12
282026-07-02T03:44:28.916ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14271, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:31159 req_id = d59c26d6-16bf-4302-b981-c83ff3e2731b
292026-07-02T03:44:28.952ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2854, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:59393 req_id = 6de94591-a3e4-4b9e-9bdd-343b93a3d84a
302026-07-02T03:44:29.317ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41143, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63788 req_id = 52fc669a-5106-47b0-9a9b-66ef60ddbb33
312026-07-02T03:44:29.559ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25589, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:60620 req_id = c539f8e8-e2f5-4fc4-a444-fcbe04edf5f0
322026-07-02T03:44:39.319ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25042, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53793 req_id = b1d8a832-04a7-405f-8be6-85b1c12ba82d
332026-07-02T03:44:49.321ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1700, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55147 req_id = 882184fc-06ac-46ba-82e9-968d0ed04092
342026-07-02T03:44:59.323ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2429, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59199 req_id = fcb85e61-0f8c-44f3-aca2-8b7e689d96c3
352026-07-02T03:45:00.292ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23838, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_dcb336db-e940-41be-8947-64e008af2100."), ), original: Query { name: Name("oxz_ntp_dcb336db-e940-41be-8947-64e008af2100."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 100, 99, 98, 51, 51, 54, 100, 98, 45, 101, 57, 52, 48, 45, 52, 49, 98, 101, 45, 56, 57, 52, 55, 45, 54, 52, 101, 48, 48, 56, 97, 102, 50, 49, 48, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_dcb336db-e940-41be-8947-64e008af2100." peer_addr = [fd00:1122:3344:101::1b]:63842 req_id = 8d33694e-e2a3-42ce-a2d3-599b5e35f594
362026-07-02T03:45:00.308ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:34373 req_id = 5379a90f-d0c4-4287-bc82-0c5d13b46822
372026-07-02T03:45:00.309ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:56172 req_id = 19cfd9ec-a993-4d54-8650-fa797b32d1bf
382026-07-02T03:45:00.463ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9321, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:42913 req_id = 85fe31f5-a5fa-4d9e-97ed-b2456a0feef0
392026-07-02T03:45:00.464ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9321, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:54371 req_id = b042fc84-2d5c-4826-bac0-c50d7b9ef791
402026-07-02T03:45:00.576ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45455, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), ), original: Query { name: Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 55, 49, 48, 51, 98, 51, 57, 45, 48, 99, 102, 53, 45, 52, 99, 56, 54, 45, 56, 100, 57, 100, 45, 100, 48, 50, 99, 52, 51, 55, 56, 101, 53, 55, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576." peer_addr = [fd00:1122:3344:101::4]:33432 req_id = f23e6c82-29db-4501-8f95-08ef147d7303
412026-07-02T03:45:00.578ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45455, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), ), original: Query { name: Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 55, 49, 48, 51, 98, 51, 57, 45, 48, 99, 102, 53, 45, 52, 99, 56, 54, 45, 56, 100, 57, 100, 45, 100, 48, 50, 99, 52, 51, 55, 56, 101, 53, 55, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576." peer_addr = [fd00:1122:3344:101::4]:41165 req_id = d0c176f1-0e69-4acb-9cc8-d9e382cc1897
422026-07-02T03:45:00.653ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), ), original: Query { name: Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 55, 55, 57, 48, 57, 48, 99, 45, 102, 48, 97, 57, 45, 52, 98, 55, 52, 45, 57, 99, 48, 48, 45, 51, 53, 100, 101, 53, 51, 51, 101, 98, 49, 52, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d." peer_addr = [fd00:1122:3344:101::5]:64370 req_id = fa967b01-2ee6-4f0e-9ae8-810dbad8e48f
432026-07-02T03:45:00.654ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), ), original: Query { name: Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 55, 55, 57, 48, 57, 48, 99, 45, 102, 48, 97, 57, 45, 52, 98, 55, 52, 45, 57, 99, 48, 48, 45, 51, 53, 100, 101, 53, 51, 51, 101, 98, 49, 52, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d." peer_addr = [fd00:1122:3344:101::5]:47054 req_id = fefa8942-2b20-48c4-96ed-f0ec2987eab5
442026-07-02T03:45:00.680ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49561, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), ), original: Query { name: Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 102, 52, 50, 101, 50, 50, 50, 45, 52, 100, 100, 54, 45, 52, 48, 99, 52, 45, 56, 48, 57, 52, 45, 52, 48, 54, 98, 102, 100, 57, 55, 101, 51, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394." peer_addr = [fd00:1122:3344:101::3]:49878 req_id = 75aedeb1-dbdc-4b37-a60d-17d94fa6acef
452026-07-02T03:45:00.681ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49561, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), ), original: Query { name: Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 102, 52, 50, 101, 50, 50, 50, 45, 52, 100, 100, 54, 45, 52, 48, 99, 52, 45, 56, 48, 57, 52, 45, 52, 48, 54, 98, 102, 100, 57, 55, 101, 51, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394." peer_addr = [fd00:1122:3344:101::3]:44096 req_id = 7a856ec4-200c-4c20-a304-be8967162c64
462026-07-02T03:45:00.684ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10081, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), ), original: Query { name: Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 50, 54, 100, 54, 49, 101, 53, 45, 102, 101, 51, 98, 45, 52, 51, 51, 100, 45, 57, 53, 50, 48, 45, 54, 48, 48, 101, 99, 102, 53, 48, 57, 49, 48, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f." peer_addr = [fd00:1122:3344:101::6]:36625 req_id = f9f45451-bd24-458a-b9fd-1161db58fb87
472026-07-02T03:45:00.684ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10081, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), ), original: Query { name: Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 50, 54, 100, 54, 49, 101, 53, 45, 102, 101, 51, 98, 45, 52, 51, 51, 100, 45, 57, 53, 50, 48, 45, 54, 48, 48, 101, 99, 102, 53, 48, 57, 49, 48, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f." peer_addr = [fd00:1122:3344:101::6]:37379 req_id = 12d140cc-94b7-46c6-9567-0de5e4f58588
482026-07-02T03:45:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7086, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), ), original: Query { name: Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 100, 100, 101, 53, 50, 102, 49, 45, 102, 99, 49, 49, 45, 52, 49, 101, 55, 45, 98, 49, 56, 98, 45, 102, 49, 53, 49, 48, 57, 98, 56, 99, 49, 56, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186." peer_addr = [fd00:1122:3344:101::7]:46865 req_id = d8e16033-90e4-4180-b58d-5d88db3462c8
492026-07-02T03:45:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7086, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), ), original: Query { name: Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 100, 100, 101, 53, 50, 102, 49, 45, 102, 99, 49, 49, 45, 52, 49, 101, 55, 45, 98, 49, 56, 98, 45, 102, 49, 53, 49, 48, 57, 98, 56, 99, 49, 56, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186." peer_addr = [fd00:1122:3344:101::7]:35144 req_id = 1e1ad992-1004-43a2-89e4-6ce3e756a94f