Max level shown:
1 [ Aug 28 19:28:34 Enabled. ]
2 [ Aug 28 19:28:34 Rereading configuration. ]
3 [ Aug 28 19:28:34 Rereading configuration. ]
4 [ Aug 28 19:28:36 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Aug 28 19:28:36 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-08-28T19:28:36.019ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-08-28T19:28:36.019ZINFOdns-server (store): opening sled database path = /data/dns
102026-08-28T19:28:36.041ZINFOdns-server (store): pruning trees for generations newer than 0
112026-08-28T19:28:36.041ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-08-28T19:28:36.041ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-08-28T19:28:36.041ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-08-28T19:28:36.462ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17328, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:32796 req_id = 33b6add9-98e2-4c81-bb11-44e9a4ea436f
152026-08-28T19:28:36.468ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17355, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:47041 req_id = 5046598f-4005-4234-aa83-71794c7b8490
162026-08-28T19:28:36.574ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56815, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:18378 req_id = dd3f9ab9-ea10-4218-b704-3d67fa2563cf
172026-08-28T19:28:36.700ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57168, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:42823 req_id = 8fe66534-746c-49a5-800e-ad5bd321b2ea
182026-08-28T19:28:36.796ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63559, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:34317 req_id = c28d3319-27d5-42f9-adac-a22ae8b2463f
192026-08-28T19:28:36.813ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31586, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:34792 req_id = cbf99feb-765e-4f68-9bef-093e65a7db23
202026-08-28T19:28:36.973ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46379, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:65113 req_id = b157e7a9-d863-43e2-8971-c5db95abe13c
212026-08-28T19:28:37.039ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17105, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:56205 req_id = 015b1f57-050c-4bdf-b5c7-265f5bed6ff0
222026-08-28T19:28:37.066ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13125, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:48654 req_id = b1ed5575-fb0e-490c-ab58-2c91f43b51f3
232026-08-28T19:28:37.279ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:9307 req_id = b40e4bc0-45af-4f83-b241-d7f3e01535dd
242026-08-28T19:28:37.363ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51144, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:45598 req_id = 7255b172-387c-4bd4-877e-509db6673cd8
252026-08-28T19:28:37.406ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6820, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:2::2]:58269 req_id = 1f45cca0-b284-4cde-b4b2-d68f2ec67835
262026-08-28T19:28:37.446ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64344, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:33733 req_id = 4823915c-d1ba-4995-94fe-3ff6118d46bc
272026-08-28T19:28:37.611ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39912, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:61301 req_id = 000bcf00-9d85-4f9a-b67a-ed7a52a69e33
282026-08-28T19:28:37.715ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25995, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:101::2]:55876 req_id = c3cf126d-8e3a-4e65-ab60-014c4b209407
292026-08-28T19:28:38.038ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:42267
302026-08-28T19:28:38.038ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = cea3f6ce-af24-4136-a52d-080f222edf1e
312026-08-28T19:28:38.038ZINFOdns-server (store): pruning trees for generations newer than 0
322026-08-28T19:28:38.051ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
332026-08-28T19:28:38.051ZINFOdns-server (store): updated generation new_generation = 1 req_id = cea3f6ce-af24-4136-a52d-080f222edf1e
342026-08-28T19:28:38.051ZINFOdns-server (http): request completed latency_us = 13159 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:42267 req_id = cea3f6ce-af24-4136-a52d-080f222edf1e response_code = 204 uri = /config
352026-08-28T19:28:49.242ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64573, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:55681 req_id = 9b50672c-2bdc-4b09-a514-94eca188f1a6
362026-08-28T19:28:49.247ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1952, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:56955 req_id = 6c7f8cf1-622a-4a08-a732-11691a66bf17
372026-08-28T19:28:49.251ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19481, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:38055 req_id = 01ea4ef9-ba1d-40d8-a6c8-3d9eb1a4bcea
382026-08-28T19:29:44.029ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30533, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49968 req_id = 0413a31d-99b3-491a-916e-a9258cb8f6a5
392026-08-28T19:29:50.388ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 400, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), ), original: Query { name: Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 97, 98, 99, 99, 97, 98, 100, 45, 99, 57, 99, 100, 45, 52, 54, 50, 49, 45, 98, 100, 101, 101, 45, 56, 57, 99, 98, 101, 101, 100, 56, 51, 56, 102, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7." peer_addr = [fd00:1122:3344:101::e]:44573 req_id = 5670717e-a7c3-470b-bfd9-5b266db3259e
402026-08-28T19:29:50.389ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 400, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), ), original: Query { name: Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 97, 98, 99, 99, 97, 98, 100, 45, 99, 57, 99, 100, 45, 52, 54, 50, 49, 45, 98, 100, 101, 101, 45, 56, 57, 99, 98, 101, 101, 100, 56, 51, 56, 102, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7." peer_addr = [fd00:1122:3344:101::e]:41626 req_id = ffd1e29f-f5bf-456c-8b33-99bcf0da0224
412026-08-28T19:29:54.031ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49892, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56124 req_id = a069ac1c-faca-48a1-82e2-9ba08b1baeea
422026-08-28T19:29:57.542ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55645, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:51583 req_id = 64eccb97-3b8e-47bd-9195-a8804488723c
432026-08-28T19:29:57.565ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9551, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:50048 req_id = b8f5d33b-801c-45eb-b0dc-9ef01465c68b
442026-08-28T19:29:57.845ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33071, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:9693 req_id = b2528ee9-ffdd-49dd-8e24-9c6797326487
452026-08-28T19:29:58.715ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:35137
462026-08-28T19:29:58.716ZINFOdns-server (http): request completed latency_us = 394 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:35137 req_id = 8d3d6d69-b6e8-45d5-8241-e2b946a19233 response_code = 200 uri = /config
472026-08-28T19:29:59.659ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58059, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:39308 req_id = bcd6aaa9-6d90-476e-8174-a48652d44dca
482026-08-28T19:29:59.997ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:63323
492026-08-28T19:29:59.997ZINFOdns-server (http): request completed latency_us = 374 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:63323 req_id = 9d59437b-cc30-4b2e-bfd4-2f43b166f698 response_code = 200 uri = /config
502026-08-28T19:30:00.008ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:39894
512026-08-28T19:30:00.011ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 5d60a7cd-f010-4eb2-b88b-816e3bd99e04
522026-08-28T19:30:00.011ZINFOdns-server (store): updated generation new_generation = 1 req_id = 5d60a7cd-f010-4eb2-b88b-816e3bd99e04
532026-08-28T19:30:00.011ZINFOdns-server (http): request completed latency_us = 306 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:39894 req_id = 5d60a7cd-f010-4eb2-b88b-816e3bd99e04 response_code = 204 uri = /config
542026-08-28T19:30:00.038ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2125, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_94f81991-4dc0-48d5-bb02-c3419ea03207."), ), original: Query { name: Name("oxz_ntp_94f81991-4dc0-48d5-bb02-c3419ea03207."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 57, 52, 102, 56, 49, 57, 57, 49, 45, 52, 100, 99, 48, 45, 52, 56, 100, 53, 45, 98, 98, 48, 50, 45, 99, 51, 52, 49, 57, 101, 97, 48, 51, 50, 48, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_94f81991-4dc0-48d5-bb02-c3419ea03207." peer_addr = [fd00:1122:3344:101::1b]:39328 req_id = 028ed97f-e856-4ca1-a14c-ae9b35661de3
552026-08-28T19:30:00.142ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60852, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), ), original: Query { name: Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 97, 98, 99, 99, 97, 98, 100, 45, 99, 57, 99, 100, 45, 52, 54, 50, 49, 45, 98, 100, 101, 101, 45, 56, 57, 99, 98, 101, 101, 100, 56, 51, 56, 102, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7." peer_addr = [fd00:1122:3344:101::e]:59174 req_id = f0bf5c2d-d929-46fd-91b2-d59bea5355b5
562026-08-28T19:30:00.143ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60852, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), ), original: Query { name: Name("oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 99, 97, 98, 99, 99, 97, 98, 100, 45, 99, 57, 99, 100, 45, 52, 54, 50, 49, 45, 98, 100, 101, 101, 45, 56, 57, 99, 98, 101, 101, 100, 56, 51, 56, 102, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_cabccabd-c9cd-4621-bdee-89cbeed838f7." peer_addr = [fd00:1122:3344:101::e]:55477 req_id = c1fce817-8de3-435f-9245-594ff38727ed
572026-08-28T19:30:00.454ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59602, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:60016 req_id = 9896280d-642c-497b-90dc-8b1712c13bf2
582026-08-28T19:30:00.455ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59602, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:51103 req_id = ed15e834-cfe2-4faa-abbc-3ee6d58efc41
592026-08-28T19:30:00.645ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52747, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d9383628-af18-4bbd-9bda-b2037d7ef9a7."), ), original: Query { name: Name("oxz_cockroachdb_d9383628-af18-4bbd-9bda-b2037d7ef9a7."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 57, 51, 56, 51, 54, 50, 56, 45, 97, 102, 49, 56, 45, 52, 98, 98, 100, 45, 57, 98, 100, 97, 45, 98, 50, 48, 51, 55, 100, 55, 101, 102, 57, 97, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d9383628-af18-4bbd-9bda-b2037d7ef9a7." peer_addr = [fd00:1122:3344:101::6]:62208 req_id = 9408c900-0664-4e9c-8842-65673affe90c
602026-08-28T19:30:00.646ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52747, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d9383628-af18-4bbd-9bda-b2037d7ef9a7."), ), original: Query { name: Name("oxz_cockroachdb_d9383628-af18-4bbd-9bda-b2037d7ef9a7."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 57, 51, 56, 51, 54, 50, 56, 45, 97, 102, 49, 56, 45, 52, 98, 98, 100, 45, 57, 98, 100, 97, 45, 98, 50, 48, 51, 55, 100, 55, 101, 102, 57, 97, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d9383628-af18-4bbd-9bda-b2037d7ef9a7." peer_addr = [fd00:1122:3344:101::6]:44332 req_id = 1ede198e-1077-45e9-ac9e-05eb6ad55b3d
612026-08-28T19:30:00.678ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16403, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a883b6e3-0fc2-43ae-a3c3-1351845542b5."), ), original: Query { name: Name("oxz_cockroachdb_a883b6e3-0fc2-43ae-a3c3-1351845542b5."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 56, 56, 51, 98, 54, 101, 51, 45, 48, 102, 99, 50, 45, 52, 51, 97, 101, 45, 97, 51, 99, 51, 45, 49, 51, 53, 49, 56, 52, 53, 53, 52, 50, 98, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a883b6e3-0fc2-43ae-a3c3-1351845542b5." peer_addr = [fd00:1122:3344:101::7]:46158 req_id = 34da2f03-f998-484a-879f-adfdfd878ed0
622026-08-28T19:30:00.678ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16403, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_a883b6e3-0fc2-43ae-a3c3-1351845542b5."), ), original: Query { name: Name("oxz_cockroachdb_a883b6e3-0fc2-43ae-a3c3-1351845542b5."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 97, 56, 56, 51, 98, 54, 101, 51, 45, 48, 102, 99, 50, 45, 52, 51, 97, 101, 45, 97, 51, 99, 51, 45, 49, 51, 53, 49, 56, 52, 53, 53, 52, 50, 98, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_a883b6e3-0fc2-43ae-a3c3-1351845542b5." peer_addr = [fd00:1122:3344:101::7]:39142 req_id = 751e5f83-c303-40f5-97a9-bfaff4c69bd8
632026-08-28T19:30:00.786ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46319, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:18157 req_id = 3dfded10-68b8-4dc6-954f-5dcd8659bfae
642026-08-28T19:30:00.921ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18993, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_668e223c-3348-48a5-9d9a-aa928ad0f4da."), ), original: Query { name: Name("oxz_cockroachdb_668e223c-3348-48a5-9d9a-aa928ad0f4da."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 54, 56, 101, 50, 50, 51, 99, 45, 51, 51, 52, 56, 45, 52, 56, 97, 53, 45, 57, 100, 57, 97, 45, 97, 97, 57, 50, 56, 97, 100, 48, 102, 52, 100, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_668e223c-3348-48a5-9d9a-aa928ad0f4da." peer_addr = [fd00:1122:3344:101::4]:36868 req_id = 853d67c4-0981-40ee-b913-e251923a05a4
652026-08-28T19:30:00.921ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18993, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_668e223c-3348-48a5-9d9a-aa928ad0f4da."), ), original: Query { name: Name("oxz_cockroachdb_668e223c-3348-48a5-9d9a-aa928ad0f4da."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 54, 56, 101, 50, 50, 51, 99, 45, 51, 51, 52, 56, 45, 52, 56, 97, 53, 45, 57, 100, 57, 97, 45, 97, 97, 57, 50, 56, 97, 100, 48, 102, 52, 100, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_668e223c-3348-48a5-9d9a-aa928ad0f4da." peer_addr = [fd00:1122:3344:101::4]:38272 req_id = 91484d9a-1a85-4c54-b32b-0033f8dfa024
662026-08-28T19:30:00.958ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48755, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_bd118dd1-0c87-4170-94f9-a40510cd46d1."), ), original: Query { name: Name("oxz_cockroachdb_bd118dd1-0c87-4170-94f9-a40510cd46d1."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 100, 49, 49, 56, 100, 100, 49, 45, 48, 99, 56, 55, 45, 52, 49, 55, 48, 45, 57, 52, 102, 57, 45, 97, 52, 48, 53, 49, 48, 99, 100, 52, 54, 100, 49, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_bd118dd1-0c87-4170-94f9-a40510cd46d1." peer_addr = [fd00:1122:3344:101::3]:63673 req_id = f2262c02-af54-45a9-ae71-ff59c50e3f53
672026-08-28T19:30:00.959ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48755, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_bd118dd1-0c87-4170-94f9-a40510cd46d1."), ), original: Query { name: Name("oxz_cockroachdb_bd118dd1-0c87-4170-94f9-a40510cd46d1."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 100, 49, 49, 56, 100, 100, 49, 45, 48, 99, 56, 55, 45, 52, 49, 55, 48, 45, 57, 52, 102, 57, 45, 97, 52, 48, 53, 49, 48, 99, 100, 52, 54, 100, 49, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_bd118dd1-0c87-4170-94f9-a40510cd46d1." peer_addr = [fd00:1122:3344:101::3]:40328 req_id = 085caedb-6d62-4c83-8e0f-fd428b22a869
682026-08-28T19:30:00.960ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12108, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_17b674d0-e1f0-47fb-9b1d-41531d93274a."), ), original: Query { name: Name("oxz_cockroachdb_17b674d0-e1f0-47fb-9b1d-41531d93274a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 55, 98, 54, 55, 52, 100, 48, 45, 101, 49, 102, 48, 45, 52, 55, 102, 98, 45, 57, 98, 49, 100, 45, 52, 49, 53, 51, 49, 100, 57, 51, 50, 55, 52, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_17b674d0-e1f0-47fb-9b1d-41531d93274a." peer_addr = [fd00:1122:3344:101::5]:55946 req_id = f1075314-b76a-4745-8c89-2a2f5e29cb37
692026-08-28T19:30:00.961ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12108, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_17b674d0-e1f0-47fb-9b1d-41531d93274a."), ), original: Query { name: Name("oxz_cockroachdb_17b674d0-e1f0-47fb-9b1d-41531d93274a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 55, 98, 54, 55, 52, 100, 48, 45, 101, 49, 102, 48, 45, 52, 55, 102, 98, 45, 57, 98, 49, 100, 45, 52, 49, 53, 51, 49, 100, 57, 51, 50, 55, 52, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_17b674d0-e1f0-47fb-9b1d-41531d93274a." peer_addr = [fd00:1122:3344:101::5]:38981 req_id = d4ea86a0-2052-4004-808d-5569ec85e0a6
702026-08-28T19:30:00.985ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:56595
712026-08-28T19:30:00.986ZINFOdns-server (http): request completed latency_us = 279 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:56595 req_id = 56c5ddfe-eef5-439d-907e-9803e5bc4c58 response_code = 200 uri = /config
722026-08-28T19:30:01.986ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20407, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:4368 req_id = f069d3f3-28a4-4cd4-b088-c28e6ec26727
732026-08-28T19:30:02.139ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:51753
742026-08-28T19:30:02.139ZINFOdns-server (http): request completed latency_us = 278 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:51753 req_id = 97f08639-5060-4f13-ae44-d977c32e54bc response_code = 200 uri = /config