Max level shown:
1 [ Jul 2 03:42:52 Enabled. ]
2 [ Jul 2 03:42:52 Rereading configuration. ]
3 [ Jul 2 03:42:53 Rereading configuration. ]
4 [ Jul 2 03:42:54 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Jul 2 03:42:54 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-02T03:42:54.510ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-07-02T03:42:54.510ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-02T03:42:54.533ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-02T03:42:54.533ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-02T03:42:54.533ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-07-02T03:42:54.533ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-07-02T03:42:54.671ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8279, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:35564 req_id = e492897d-7c9c-4533-8ede-f8ccbdd7ac33
152026-07-02T03:42:54.846ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30910, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:60539 req_id = e87a7288-c366-4de3-b824-6ff221b65ea8
162026-07-02T03:42:55.091ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:63921
172026-07-02T03:42:55.091ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = f82f4598-aa33-40a1-9f3b-36da50860e4d
182026-07-02T03:42:55.091ZINFOdns-server (store): pruning trees for generations newer than 0
192026-07-02T03:42:55.104ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
202026-07-02T03:42:55.104ZINFOdns-server (store): updated generation new_generation = 1 req_id = f82f4598-aa33-40a1-9f3b-36da50860e4d
212026-07-02T03:42:55.104ZINFOdns-server (http): request completed latency_us = 13306 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:63921 req_id = f82f4598-aa33-40a1-9f3b-36da50860e4d response_code = 204 uri = /config
222026-07-02T03:43:07.923ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13904, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:42584 req_id = 8dba7ba0-f445-4aea-9207-3ecef6830c20
232026-07-02T03:43:07.928ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45084, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:63832 req_id = f86d46bf-ceaf-4f9f-bc71-3a87f6c9f807
242026-07-02T03:43:07.933ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39414, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:46016 req_id = 91123ab5-b72d-4237-8b5e-8b08cdaf881f
252026-07-02T03:44:19.333ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56879, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60024 req_id = 954ed607-4615-48be-939e-549bff39b232
262026-07-02T03:44:20.961ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:47562 req_id = 1c8b1330-9adc-4d49-8ee3-9f6c4433d405
272026-07-02T03:44:20.964ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30376, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:54951 req_id = bb89d046-a633-48f6-9e87-686aa9a91418
282026-07-02T03:44:28.916ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37156, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:45075 req_id = fc3c68e5-3ed3-4650-aee9-ae18192a621e
292026-07-02T03:44:28.952ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4886, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:45291 req_id = ede847af-b71d-4254-b441-dd2c1b1f5989
302026-07-02T03:44:29.317ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24274, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60213 req_id = c083bb29-1937-4a95-b0d7-1aa22a3b75fa
312026-07-02T03:44:29.559ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56049, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:24608 req_id = eb01db23-0635-4ea8-9d4e-c0565cc2a60f
322026-07-02T03:44:39.319ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60289, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54133 req_id = 2ea192fb-e7ed-41be-9e61-4a5ff6509165
332026-07-02T03:44:49.321ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52988, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55558 req_id = 239239a5-2fb2-4fa8-9a61-e3b57847ac74
342026-07-02T03:44:59.323ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2005, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62834 req_id = 11231aae-92fb-404d-9379-ec12f4eb8f8b
352026-07-02T03:45:00.292ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23838, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_dcb336db-e940-41be-8947-64e008af2100."), ), original: Query { name: Name("oxz_ntp_dcb336db-e940-41be-8947-64e008af2100."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 100, 99, 98, 51, 51, 54, 100, 98, 45, 101, 57, 52, 48, 45, 52, 49, 98, 101, 45, 56, 57, 52, 55, 45, 54, 52, 101, 48, 48, 56, 97, 102, 50, 49, 48, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_dcb336db-e940-41be-8947-64e008af2100." peer_addr = [fd00:1122:3344:101::1b]:43288 req_id = 51988a18-e0c4-45bd-b14c-08a1c87e5c55
362026-07-02T03:45:00.309ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:59055 req_id = f25c8d9a-1b9e-42ac-aa51-b2cf6f9e0e0f
372026-07-02T03:45:00.309ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:44328 req_id = a8b53da3-7581-497f-b618-d573e0e69b9f
382026-07-02T03:45:00.463ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9321, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:58712 req_id = e0141a13-74dc-41bd-ab00-649fdcd254a5
392026-07-02T03:45:00.464ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9321, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), ), original: Query { name: Name("oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 55, 54, 52, 55, 55, 100, 53, 45, 50, 53, 99, 56, 45, 52, 48, 102, 101, 45, 56, 55, 53, 51, 45, 56, 50, 100, 102, 49, 57, 54, 57, 97, 98, 98, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_c76477d5-25c8-40fe-8753-82df1969abbc." peer_addr = [fd00:1122:3344:101::e]:35057 req_id = b8aae569-05d2-4be9-b731-5befbfa66c79
402026-07-02T03:45:00.577ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45455, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), ), original: Query { name: Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 55, 49, 48, 51, 98, 51, 57, 45, 48, 99, 102, 53, 45, 52, 99, 56, 54, 45, 56, 100, 57, 100, 45, 100, 48, 50, 99, 52, 51, 55, 56, 101, 53, 55, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576." peer_addr = [fd00:1122:3344:101::4]:64875 req_id = b11f9195-1df2-4fa9-97b7-edc401b89499
412026-07-02T03:45:00.578ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45455, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), ), original: Query { name: Name("oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 55, 49, 48, 51, 98, 51, 57, 45, 48, 99, 102, 53, 45, 52, 99, 56, 54, 45, 56, 100, 57, 100, 45, 100, 48, 50, 99, 52, 51, 55, 56, 101, 53, 55, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_c7103b39-0cf5-4c86-8d9d-d02c4378e576." peer_addr = [fd00:1122:3344:101::4]:35500 req_id = be50b119-8622-48ec-8edf-a4eff76e4fb5
422026-07-02T03:45:00.653ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), ), original: Query { name: Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 55, 55, 57, 48, 57, 48, 99, 45, 102, 48, 97, 57, 45, 52, 98, 55, 52, 45, 57, 99, 48, 48, 45, 51, 53, 100, 101, 53, 51, 51, 101, 98, 49, 52, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d." peer_addr = [fd00:1122:3344:101::5]:33979 req_id = 01539c67-a954-4044-9af0-bf5b260eb16b
432026-07-02T03:45:00.654ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7604, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), ), original: Query { name: Name("oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 101, 55, 55, 57, 48, 57, 48, 99, 45, 102, 48, 97, 57, 45, 52, 98, 55, 52, 45, 57, 99, 48, 48, 45, 51, 53, 100, 101, 53, 51, 51, 101, 98, 49, 52, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_e779090c-f0a9-4b74-9c00-35de533eb14d." peer_addr = [fd00:1122:3344:101::5]:36174 req_id = 0920d300-86c9-4dbb-b483-2c721b4bfd95
442026-07-02T03:45:00.681ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49561, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), ), original: Query { name: Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 102, 52, 50, 101, 50, 50, 50, 45, 52, 100, 100, 54, 45, 52, 48, 99, 52, 45, 56, 48, 57, 52, 45, 52, 48, 54, 98, 102, 100, 57, 55, 101, 51, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394." peer_addr = [fd00:1122:3344:101::3]:54022 req_id = ac785237-60b6-4fc8-a0c2-d97a79222c0d
452026-07-02T03:45:00.682ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49561, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), ), original: Query { name: Name("oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 102, 52, 50, 101, 50, 50, 50, 45, 52, 100, 100, 54, 45, 52, 48, 99, 52, 45, 56, 48, 57, 52, 45, 52, 48, 54, 98, 102, 100, 57, 55, 101, 51, 57, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7f42e222-4dd6-40c4-8094-406bfd97e394." peer_addr = [fd00:1122:3344:101::3]:56771 req_id = fd63662b-8e37-4781-ab25-4315d8e9bc69
462026-07-02T03:45:00.684ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10081, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), ), original: Query { name: Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 50, 54, 100, 54, 49, 101, 53, 45, 102, 101, 51, 98, 45, 52, 51, 51, 100, 45, 57, 53, 50, 48, 45, 54, 48, 48, 101, 99, 102, 53, 48, 57, 49, 48, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f." peer_addr = [fd00:1122:3344:101::6]:64854 req_id = ef4efe6c-5179-48fd-aeb8-98adc6d80eb5
472026-07-02T03:45:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10081, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), ), original: Query { name: Name("oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 50, 54, 100, 54, 49, 101, 53, 45, 102, 101, 51, 98, 45, 52, 51, 51, 100, 45, 57, 53, 50, 48, 45, 54, 48, 48, 101, 99, 102, 53, 48, 57, 49, 48, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a26d61e5-fe3b-433d-9520-600ecf50910f." peer_addr = [fd00:1122:3344:101::6]:43010 req_id = a0360c81-4799-47ed-b02d-ab89c6ccc436
482026-07-02T03:45:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7086, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), ), original: Query { name: Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 100, 100, 101, 53, 50, 102, 49, 45, 102, 99, 49, 49, 45, 52, 49, 101, 55, 45, 98, 49, 56, 98, 45, 102, 49, 53, 49, 48, 57, 98, 56, 99, 49, 56, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186." peer_addr = [fd00:1122:3344:101::7]:52766 req_id = b0448265-6788-4942-a362-790185fb7136
492026-07-02T03:45:00.685ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7086, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), ), original: Query { name: Name("oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 53, 100, 100, 101, 53, 50, 102, 49, 45, 102, 99, 49, 49, 45, 52, 49, 101, 55, 45, 98, 49, 56, 98, 45, 102, 49, 53, 49, 48, 57, 98, 56, 99, 49, 56, 54, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_5dde52f1-fc11-41e7-b18b-f15109b8c186." peer_addr = [fd00:1122:3344:101::7]:42932 req_id = 76c15371-9229-42a9-a3d9-4c3544dbb203