Max level shown:
1 [ Aug 29 02:57:32 Enabled. ]
2 [ Aug 29 02:57:32 Rereading configuration. ]
3 [ Aug 29 02:57:33 Rereading configuration. ]
4 [ Aug 29 02:57:34 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Aug 29 02:57:34 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-08-29T02:57:34.274ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-08-29T02:57:34.274ZINFOdns-server (store): opening sled database path = /data/dns
102026-08-29T02:57:34.292ZINFOdns-server (store): pruning trees for generations newer than 0
112026-08-29T02:57:34.292ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-08-29T02:57:34.292ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-08-29T02:57:34.292ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-08-29T02:57:34.401ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:39541
152026-08-29T02:57:34.402ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = db0f542a-7c5c-4442-8d23-cd1faf8d45b0
162026-08-29T02:57:34.402ZINFOdns-server (store): pruning trees for generations newer than 0
172026-08-29T02:57:34.414ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-08-29T02:57:34.414ZINFOdns-server (store): updated generation new_generation = 1 req_id = db0f542a-7c5c-4442-8d23-cd1faf8d45b0
192026-08-29T02:57:34.414ZINFOdns-server (http): request completed latency_us = 12924 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:39541 req_id = db0f542a-7c5c-4442-8d23-cd1faf8d45b0 response_code = 204 uri = /config
202026-08-29T02:57:46.514ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 760, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:44361 req_id = e35bbb5d-d5c7-4f49-91f8-a6038626d755
212026-08-29T02:57:46.518ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34679, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:55377 req_id = 1cde6a12-1ee2-4dcd-9738-f69f20fe0108
222026-08-29T02:57:46.522ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29461, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:39020 req_id = 2fcedc71-8144-4d51-94fd-a569fc981352
232026-08-29T02:59:28.750ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8589, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61954 req_id = 0a388096-01df-45bc-8cf0-572573cb5c2a
242026-08-29T02:59:32.970ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25225, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), ), original: Query { name: Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 100, 49, 51, 48, 102, 48, 56, 98, 45, 57, 53, 50, 53, 45, 52, 49, 101, 56, 45, 98, 50, 97, 97, 45, 57, 50, 99, 48, 51, 97, 101, 53, 52, 98, 52, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44." peer_addr = [fd00:1122:3344:101::e]:56848 req_id = 4d11af72-50d2-4314-ba18-1f1b417d7ca5
252026-08-29T02:59:32.970ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25225, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), ), original: Query { name: Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 100, 49, 51, 48, 102, 48, 56, 98, 45, 57, 53, 50, 53, 45, 52, 49, 101, 56, 45, 98, 50, 97, 97, 45, 57, 50, 99, 48, 51, 97, 101, 53, 52, 98, 52, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44." peer_addr = [fd00:1122:3344:101::e]:43886 req_id = fcc14b2c-38d0-41a7-9cec-8553ffae087a
262026-08-29T02:59:38.734ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41705, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57120 req_id = 3c67100a-b5b6-4f1c-bf00-a6ea9bb87785
272026-08-29T02:59:41.638ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46959, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:38502 req_id = 43133e31-bd84-40d5-891b-7599534d81fb
282026-08-29T02:59:41.876ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50426, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:52669 req_id = 2ceed37a-a2dd-45fd-8f7b-fb554e7bfe24
292026-08-29T02:59:43.100ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38481, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:62964 req_id = 149a7276-b156-4697-9de3-90fec66d3847
302026-08-29T02:59:48.736ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38150, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62027 req_id = 807addfd-82c5-4edd-819b-12203bf1b632
312026-08-29T02:59:55.462ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37029, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:15457 req_id = 7bbbfe8f-b207-43c6-ba13-b71e94c6d570
322026-08-29T02:59:55.823ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:46017
332026-08-29T02:59:55.910ZINFOdns-server (http): request completed latency_us = 22531 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:46017 req_id = 10da528c-5b0b-4a24-8bd8-b6fb62d832ea response_code = 200 uri = /config
342026-08-29T02:59:58.739ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58272, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59160 req_id = 2170719e-e98d-46c7-963b-5c1c55503bec
352026-08-29T02:59:59.110ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52900, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:14586 req_id = 57b3250c-6f19-45b8-9864-cb870653ed0e
362026-08-29T02:59:59.126ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:60455
372026-08-29T02:59:59.126ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = c6d0bf6b-4f7c-46d0-959e-4c1eeb638656
382026-08-29T02:59:59.126ZINFOdns-server (store): updated generation new_generation = 1 req_id = c6d0bf6b-4f7c-46d0-959e-4c1eeb638656
392026-08-29T02:59:59.126ZINFOdns-server (http): request completed latency_us = 276 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:60455 req_id = c6d0bf6b-4f7c-46d0-959e-4c1eeb638656 response_code = 204 uri = /config
402026-08-29T02:59:59.248ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:34642
412026-08-29T02:59:59.249ZINFOdns-server (http): request completed latency_us = 218 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:34642 req_id = a0aa3f9c-a0b5-4899-951f-a953c9186a3a response_code = 200 uri = /config
422026-08-29T03:00:00.252ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16951, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:50072 req_id = 5e9b200e-3456-42d3-8058-9ffb702daef5
432026-08-29T03:00:00.392ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:62853
442026-08-29T03:00:00.392ZINFOdns-server (http): request completed latency_us = 246 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:62853 req_id = 6ba0023b-cb00-44df-aa5b-b2d4fa2a2d1c response_code = 200 uri = /config
452026-08-29T03:00:00.809ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29070, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), ), original: Query { name: Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 100, 49, 51, 48, 102, 48, 56, 98, 45, 57, 53, 50, 53, 45, 52, 49, 101, 56, 45, 98, 50, 97, 97, 45, 57, 50, 99, 48, 51, 97, 101, 53, 52, 98, 52, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44." peer_addr = [fd00:1122:3344:101::e]:46170 req_id = 6dd4da94-fb84-488f-9f3a-480c95d57412
462026-08-29T03:00:00.810ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29070, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), ), original: Query { name: Name("oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 100, 49, 51, 48, 102, 48, 56, 98, 45, 57, 53, 50, 53, 45, 52, 49, 101, 56, 45, 98, 50, 97, 97, 45, 57, 50, 99, 48, 51, 97, 101, 53, 52, 98, 52, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_d130f08b-9525-41e8-b2aa-92c03ae54b44." peer_addr = [fd00:1122:3344:101::e]:45086 req_id = be7febf9-28c6-4911-93e4-73f8938e1381
472026-08-29T03:00:00.839ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25758, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_86b25c75-ff59-491c-827f-bd31c2c6c405."), ), original: Query { name: Name("oxz_cockroachdb_86b25c75-ff59-491c-827f-bd31c2c6c405."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 54, 98, 50, 53, 99, 55, 53, 45, 102, 102, 53, 57, 45, 52, 57, 49, 99, 45, 56, 50, 55, 102, 45, 98, 100, 51, 49, 99, 50, 99, 54, 99, 52, 48, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_86b25c75-ff59-491c-827f-bd31c2c6c405." peer_addr = [fd00:1122:3344:101::4]:37916 req_id = 5c1d6242-bbd4-43eb-85fa-4eb9d64dc396
482026-08-29T03:00:00.840ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25758, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_86b25c75-ff59-491c-827f-bd31c2c6c405."), ), original: Query { name: Name("oxz_cockroachdb_86b25c75-ff59-491c-827f-bd31c2c6c405."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 56, 54, 98, 50, 53, 99, 55, 53, 45, 102, 102, 53, 57, 45, 52, 57, 49, 99, 45, 56, 50, 55, 102, 45, 98, 100, 51, 49, 99, 50, 99, 54, 99, 52, 48, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_86b25c75-ff59-491c-827f-bd31c2c6c405." peer_addr = [fd00:1122:3344:101::4]:51849 req_id = 89dbd0bc-8c87-4037-b27a-7c95c4cfc055
492026-08-29T03:00:00.883ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26214, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_1bade0ed-0a3a-4133-846f-3f831a3c80bf."), ), original: Query { name: Name("oxz_cockroachdb_1bade0ed-0a3a-4133-846f-3f831a3c80bf."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 98, 97, 100, 101, 48, 101, 100, 45, 48, 97, 51, 97, 45, 52, 49, 51, 51, 45, 56, 52, 54, 102, 45, 51, 102, 56, 51, 49, 97, 51, 99, 56, 48, 98, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_1bade0ed-0a3a-4133-846f-3f831a3c80bf." peer_addr = [fd00:1122:3344:101::6]:48829 req_id = b95c909c-9644-47f2-8ff6-7d3eb5534ddf
502026-08-29T03:00:00.884ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26214, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_1bade0ed-0a3a-4133-846f-3f831a3c80bf."), ), original: Query { name: Name("oxz_cockroachdb_1bade0ed-0a3a-4133-846f-3f831a3c80bf."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 98, 97, 100, 101, 48, 101, 100, 45, 48, 97, 51, 97, 45, 52, 49, 51, 51, 45, 56, 52, 54, 102, 45, 51, 102, 56, 51, 49, 97, 51, 99, 56, 48, 98, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_1bade0ed-0a3a-4133-846f-3f831a3c80bf." peer_addr = [fd00:1122:3344:101::6]:41890 req_id = 2fa38f36-f1ec-4033-bf30-070dbcb176ce
512026-08-29T03:00:00.886ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30013, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_bd7f6f08-1ff9-463e-ab7a-690231b9ef5d."), ), original: Query { name: Name("oxz_cockroachdb_bd7f6f08-1ff9-463e-ab7a-690231b9ef5d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 100, 55, 102, 54, 102, 48, 56, 45, 49, 102, 102, 57, 45, 52, 54, 51, 101, 45, 97, 98, 55, 97, 45, 54, 57, 48, 50, 51, 49, 98, 57, 101, 102, 53, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_bd7f6f08-1ff9-463e-ab7a-690231b9ef5d." peer_addr = [fd00:1122:3344:101::5]:47764 req_id = b295c2d8-37d8-4ea9-96e7-20e7966335aa
522026-08-29T03:00:00.887ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30013, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_bd7f6f08-1ff9-463e-ab7a-690231b9ef5d."), ), original: Query { name: Name("oxz_cockroachdb_bd7f6f08-1ff9-463e-ab7a-690231b9ef5d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 100, 55, 102, 54, 102, 48, 56, 45, 49, 102, 102, 57, 45, 52, 54, 51, 101, 45, 97, 98, 55, 97, 45, 54, 57, 48, 50, 51, 49, 98, 57, 101, 102, 53, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_bd7f6f08-1ff9-463e-ab7a-690231b9ef5d." peer_addr = [fd00:1122:3344:101::5]:39449 req_id = aeef1903-5074-4b41-b27d-8b51cac488b0
532026-08-29T03:00:00.912ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14813, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_45ba777a-805b-4298-8c8b-daf4d6ec914c."), ), original: Query { name: Name("oxz_cockroachdb_45ba777a-805b-4298-8c8b-daf4d6ec914c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 52, 53, 98, 97, 55, 55, 55, 97, 45, 56, 48, 53, 98, 45, 52, 50, 57, 56, 45, 56, 99, 56, 98, 45, 100, 97, 102, 52, 100, 54, 101, 99, 57, 49, 52, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_45ba777a-805b-4298-8c8b-daf4d6ec914c." peer_addr = [fd00:1122:3344:101::7]:58762 req_id = 4b75fdb8-8fe3-40cc-937e-2f0bbe5d351c
542026-08-29T03:00:00.913ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14813, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_45ba777a-805b-4298-8c8b-daf4d6ec914c."), ), original: Query { name: Name("oxz_cockroachdb_45ba777a-805b-4298-8c8b-daf4d6ec914c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 52, 53, 98, 97, 55, 55, 55, 97, 45, 56, 48, 53, 98, 45, 52, 50, 57, 56, 45, 56, 99, 56, 98, 45, 100, 97, 102, 52, 100, 54, 101, 99, 57, 49, 52, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_45ba777a-805b-4298-8c8b-daf4d6ec914c." peer_addr = [fd00:1122:3344:101::7]:56765 req_id = 0ead2ed1-6db2-4578-8eaf-92735990da0b
552026-08-29T03:00:00.958ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2293, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7075fb48-5e14-46d3-b880-64befe4ba178."), ), original: Query { name: Name("oxz_cockroachdb_7075fb48-5e14-46d3-b880-64befe4ba178."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 48, 55, 53, 102, 98, 52, 56, 45, 53, 101, 49, 52, 45, 52, 54, 100, 51, 45, 98, 56, 56, 48, 45, 54, 52, 98, 101, 102, 101, 52, 98, 97, 49, 55, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7075fb48-5e14-46d3-b880-64befe4ba178." peer_addr = [fd00:1122:3344:101::3]:65032 req_id = 4d077946-7cf9-4e52-b611-5d30e7a3884d
562026-08-29T03:00:00.959ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2293, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7075fb48-5e14-46d3-b880-64befe4ba178."), ), original: Query { name: Name("oxz_cockroachdb_7075fb48-5e14-46d3-b880-64befe4ba178."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 48, 55, 53, 102, 98, 52, 56, 45, 53, 101, 49, 52, 45, 52, 54, 100, 51, 45, 98, 56, 56, 48, 45, 54, 52, 98, 101, 102, 101, 52, 98, 97, 49, 55, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7075fb48-5e14-46d3-b880-64befe4ba178." peer_addr = [fd00:1122:3344:101::3]:37467 req_id = 9cdaeb53-e795-466f-8f93-f5ec5f693654