Max level shown:
1 [ Jul 29 16:39:06 Enabled. ]
2 [ Jul 29 16:39:06 Rereading configuration. ]
3 [ Jul 29 16:39:06 Rereading configuration. ]
4 [ Jul 29 16:39:08 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Jul 29 16:39:08 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-29T16:39:08.197ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-07-29T16:39:08.197ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-29T16:39:08.218ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-29T16:39:08.218ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-29T16:39:08.218ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-07-29T16:39:08.219ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-07-29T16:39:08.373ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:55970
152026-07-29T16:39:08.374ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 5635dccf-2cf6-4895-8c7d-b608a2d4bd2c
162026-07-29T16:39:08.374ZINFOdns-server (store): pruning trees for generations newer than 0
172026-07-29T16:39:08.387ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-07-29T16:39:08.387ZINFOdns-server (store): updated generation new_generation = 1 req_id = 5635dccf-2cf6-4895-8c7d-b608a2d4bd2c
192026-07-29T16:39:08.387ZINFOdns-server (http): request completed latency_us = 13012 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:55970 req_id = 5635dccf-2cf6-4895-8c7d-b608a2d4bd2c response_code = 204 uri = /config
202026-07-29T16:39:20.696ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57074, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:41659 req_id = bd1ec1d1-ad88-48e7-871f-885c7c070eb0
212026-07-29T16:39:20.701ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50324, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:60754 req_id = d1a56727-11bb-4a08-bf7a-add22d28a028
222026-07-29T16:39:20.705ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56411, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:33356 req_id = add1c9ef-5486-4b0c-9a64-f966676d62bc
232026-07-29T16:41:20.693ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39323, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:62766 req_id = a0b8d424-c331-4cb2-b957-2f4ed731be59
242026-07-29T16:41:20.699ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63584, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:34643 req_id = bd61a481-51c4-47fb-bd20-5bcd9f8902a3
252026-07-29T16:41:20.703ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54270, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:50978 req_id = 05a79b99-7979-4ccd-8b7b-80d3e741c16c
262026-07-29T16:42:41.384ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8466, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51427 req_id = 160c6f0d-d9c1-4fdf-a153-7ba06ed50b7a
272026-07-29T16:42:51.357ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49801, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53277 req_id = 0b969852-3d08-437f-a94d-6ab21a17cbd8
282026-07-29T16:42:54.390ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36905, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:49314 req_id = d3b50043-384e-4830-9ca8-5506f1b162ce
292026-07-29T16:42:55.233ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48115, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:4270 req_id = 73ad3940-6ec5-4158-8cd0-e425d0977099
302026-07-29T16:42:55.472ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26819, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:47905 req_id = 1f90cb7c-8b1a-4392-936d-12f9ffd82324
312026-07-29T16:42:56.665ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:49738
322026-07-29T16:42:56.670ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:36136
332026-07-29T16:42:56.726ZINFOdns-server (http): request completed latency_us = 35155 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:36136 req_id = e5c7ea1b-ea6b-4ba7-bd8e-6f221a72a10d response_code = 200 uri = /config
342026-07-29T16:42:56.726ZINFOdns-server (http): request completed latency_us = 35157 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:49738 req_id = 3ece51a9-5cd2-44de-85ca-0861ba895fc3 response_code = 200 uri = /config
352026-07-29T16:42:57.324ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30949, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:44763 req_id = 1903a1ed-4eed-4cc5-a308-51d7ee243a36
362026-07-29T16:42:57.325ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30949, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:65029 req_id = 2dbb0e77-c53c-4ac6-838e-08a20236e7c1
372026-07-29T16:42:57.998ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7706, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:33659 req_id = 4b1db1b8-050b-44f8-9dc9-a1229f491192
382026-07-29T16:42:58.225ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:39066
392026-07-29T16:42:58.225ZINFOdns-server (http): request completed latency_us = 222 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:39066 req_id = 1053ecd6-af54-48a2-88ec-cd5e3880e7e4 response_code = 200 uri = /config
402026-07-29T16:42:59.159ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14275, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:31739 req_id = b5db51b6-aa72-4ff1-8a42-5406aaafc48a
412026-07-29T16:42:59.242ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:52815
422026-07-29T16:42:59.243ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = bc975285-b91c-4149-aff5-5e483ce8e166
432026-07-29T16:42:59.243ZINFOdns-server (store): updated generation new_generation = 1 req_id = bc975285-b91c-4149-aff5-5e483ce8e166
442026-07-29T16:42:59.243ZINFOdns-server (http): request completed latency_us = 356 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:52815 req_id = bc975285-b91c-4149-aff5-5e483ce8e166 response_code = 204 uri = /config
452026-07-29T16:42:59.298ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:44988
462026-07-29T16:42:59.299ZINFOdns-server (http): request completed latency_us = 226 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:44988 req_id = a2faf8c8-1eff-4189-9e8e-f8be5813393c response_code = 200 uri = /config
472026-07-29T16:43:00.468ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33528, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:1885 req_id = d6786e86-5685-4f56-a506-28900d35c061
482026-07-29T16:43:00.594ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:62194
492026-07-29T16:43:00.594ZINFOdns-server (http): request completed latency_us = 199 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:62194 req_id = 4f0b1264-566b-4fe8-9c25-e1a5ec8bb037 response_code = 200 uri = /config
502026-07-29T16:43:01.358ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63923, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57369 req_id = 943588b9-5934-4299-9fee-152a3725345b
512026-07-29T16:43:05.523ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13044, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:5908 req_id = 77c7a039-495b-463d-9b38-665fcf2679bd
522026-07-29T16:43:05.821ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:37355
532026-07-29T16:43:05.822ZINFOdns-server (http): request completed latency_us = 222 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:37355 req_id = 906f34b2-977b-4560-973b-f3debc80e2ed response_code = 200 uri = /config
542026-07-29T16:43:06.404ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64645, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:59631 req_id = b1d4a862-3ec2-46fa-b70d-deb3d83c7d84
552026-07-29T16:43:06.515ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:63263
562026-07-29T16:43:06.516ZINFOdns-server (http): request completed latency_us = 209 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:63263 req_id = 937f8d2f-896d-4a91-bc6f-17b7a0d3c463 response_code = 200 uri = /config
572026-07-29T16:43:11.361ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56751, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54373 req_id = 9c24064a-4df5-4906-8aed-d0a45f6c42bb
582026-07-29T16:43:21.362ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42148, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64824 req_id = ea6aeb9e-455d-46bb-a0fd-479d6b102375
592026-07-29T16:43:31.363ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10312, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51925 req_id = 5e801e82-9461-4b71-9339-4ef7c9fe94e0
602026-07-29T16:43:41.366ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8955, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64174 req_id = d65dab22-3701-46f0-b22c-3f74a792bc98
612026-07-29T16:43:51.367ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54308, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56362 req_id = 106567c3-fc54-415b-a3ed-8d94a72ec2de
622026-07-29T16:43:54.208ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:57975
632026-07-29T16:43:54.210ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = e60b697d-c2d4-46f3-ab10-ab8bd8faadb5
642026-07-29T16:43:54.210ZINFOdns-server (store): updated generation new_generation = 1 req_id = e60b697d-c2d4-46f3-ab10-ab8bd8faadb5
652026-07-29T16:43:54.210ZINFOdns-server (http): request completed latency_us = 385 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:57975 req_id = e60b697d-c2d4-46f3-ab10-ab8bd8faadb5 response_code = 204 uri = /config
662026-07-29T16:43:55.102ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:52300
672026-07-29T16:43:55.104ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = e35d80f9-1c46-4744-9a97-c5e4a2415824
682026-07-29T16:43:55.104ZINFOdns-server (store): updated generation new_generation = 1 req_id = e35d80f9-1c46-4744-9a97-c5e4a2415824
692026-07-29T16:43:55.104ZINFOdns-server (http): request completed latency_us = 267 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:52300 req_id = e35d80f9-1c46-4744-9a97-c5e4a2415824 response_code = 204 uri = /config
702026-07-29T16:43:55.437ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:32949
712026-07-29T16:43:55.439ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 12d7d45e-7edb-4245-8ece-4284b6f35912
722026-07-29T16:43:55.439ZINFOdns-server (store): updated generation new_generation = 1 req_id = 12d7d45e-7edb-4245-8ece-4284b6f35912
732026-07-29T16:43:55.439ZINFOdns-server (http): request completed latency_us = 315 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:32949 req_id = 12d7d45e-7edb-4245-8ece-4284b6f35912 response_code = 204 uri = /config
742026-07-29T16:43:55.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8457, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:36994 req_id = d351cf8d-88f3-4d61-b72c-58f5ea28e21e
752026-07-29T16:43:55.745ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:50388
762026-07-29T16:43:55.745ZINFOdns-server (http): request completed latency_us = 206 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:50388 req_id = 14b26679-2964-49bc-bc83-c969fad1d3a3 response_code = 200 uri = /config
772026-07-29T16:43:56.154ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42895, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:53989 req_id = f48ae9a4-2f9e-4b83-a472-e0c93fc797ce
782026-07-29T16:43:56.258ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:50970
792026-07-29T16:43:56.258ZINFOdns-server (http): request completed latency_us = 216 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:50970 req_id = 2f2845c8-2751-40af-bfee-d8f5ffa9cc15 response_code = 200 uri = /config
802026-07-29T16:43:56.579ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15306, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:39896 req_id = 042a0a2e-1711-4958-be1b-22c9ed7de6f7
812026-07-29T16:43:56.772ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:47463
822026-07-29T16:43:56.773ZINFOdns-server (http): request completed latency_us = 193 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:47463 req_id = 797d8479-06f3-402d-bb5f-3b582b759221 response_code = 200 uri = /config
832026-07-29T16:44:01.369ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37659, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61039 req_id = a1603d98-941c-406e-9d12-ad83f7e81a27
842026-07-29T16:44:11.370ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24943, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60234 req_id = c3615cac-902a-4946-927d-4aab83fb7adc
852026-07-29T16:44:21.372ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17675, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59680 req_id = ba644a29-9c25-43f0-a843-861a8499c3ac
862026-07-29T16:44:31.375ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6108, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61163 req_id = 11cd8c51-7080-4070-a9bf-75807503c80b
872026-07-29T16:44:41.376ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40113, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:60933 req_id = cd4446b3-65f7-4d69-a759-52a1d71a9d94
882026-07-29T16:44:51.378ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52784, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63762 req_id = 96a3b9b2-4e3e-496b-9b08-886e7fae42f7
892026-07-29T16:44:54.124ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:53452
902026-07-29T16:44:54.125ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = f3f9c650-f6fd-46ad-b56a-eef7b29c2a7a
912026-07-29T16:44:54.125ZINFOdns-server (store): updated generation new_generation = 1 req_id = f3f9c650-f6fd-46ad-b56a-eef7b29c2a7a
922026-07-29T16:44:54.125ZINFOdns-server (http): request completed latency_us = 235 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:53452 req_id = f3f9c650-f6fd-46ad-b56a-eef7b29c2a7a response_code = 204 uri = /config
932026-07-29T16:44:54.895ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37225, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:61699 req_id = 85af1629-68f4-446c-98a0-46d6d319d9e6
942026-07-29T16:44:54.994ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:55079
952026-07-29T16:44:54.994ZINFOdns-server (http): request completed latency_us = 183 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:55079 req_id = a40c7cae-b289-4ebc-9cc0-4c6817f86c39 response_code = 200 uri = /config
962026-07-29T16:44:55.101ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:54398
972026-07-29T16:44:55.103ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 5a7050c8-3ead-4b7e-8d2d-fa916ebea83c
982026-07-29T16:44:55.103ZINFOdns-server (store): updated generation new_generation = 1 req_id = 5a7050c8-3ead-4b7e-8d2d-fa916ebea83c
992026-07-29T16:44:55.103ZINFOdns-server (http): request completed latency_us = 254 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:54398 req_id = 5a7050c8-3ead-4b7e-8d2d-fa916ebea83c response_code = 204 uri = /config
1002026-07-29T16:44:55.199ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:56913
1012026-07-29T16:44:55.201ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = a8967594-ef6b-4356-bf11-5943995d5fb5
1022026-07-29T16:44:55.201ZINFOdns-server (store): updated generation new_generation = 1 req_id = a8967594-ef6b-4356-bf11-5943995d5fb5
1032026-07-29T16:44:55.201ZINFOdns-server (http): request completed latency_us = 286 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:56913 req_id = a8967594-ef6b-4356-bf11-5943995d5fb5 response_code = 204 uri = /config
1042026-07-29T16:44:56.395ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50689, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:54965 req_id = 022f348f-2dba-439c-8293-cda02c07e54a
1052026-07-29T16:44:56.400ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60893, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:5230 req_id = 254c6c3b-230b-4487-aeab-88425ab3cbde
1062026-07-29T16:44:56.501ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:47840
1072026-07-29T16:44:56.502ZINFOdns-server (http): request completed latency_us = 239 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:47840 req_id = 9149d6a2-0461-45e2-aded-7b07f15b3a68 response_code = 200 uri = /config
1082026-07-29T16:44:56.592ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:44146
1092026-07-29T16:44:56.592ZINFOdns-server (http): request completed latency_us = 180 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:44146 req_id = c2de3a6a-021a-4b96-9fd8-7753623e0d56 response_code = 200 uri = /config
1102026-07-29T16:45:00.035ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), ), original: Query { name: Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 49, 57, 57, 51, 55, 54, 102, 45, 57, 102, 98, 101, 45, 52, 101, 56, 100, 45, 97, 99, 48, 102, 45, 50, 99, 49, 50, 49, 100, 48, 51, 54, 55, 54, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e." peer_addr = [fd00:1122:3344:101::3]:51651 req_id = 5caa9c18-e879-4322-90cd-1814278f50d0
1112026-07-29T16:45:00.035ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), ), original: Query { name: Name("oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 48, 49, 57, 57, 51, 55, 54, 102, 45, 57, 102, 98, 101, 45, 52, 101, 56, 100, 45, 97, 99, 48, 102, 45, 50, 99, 49, 50, 49, 100, 48, 51, 54, 55, 54, 101, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_0199376f-9fbe-4e8d-ac0f-2c121d03676e." peer_addr = [fd00:1122:3344:101::3]:35521 req_id = 3cb36ee1-cea3-4def-9c54-41de2094f976
1122026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54786, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), ), original: Query { name: Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 56, 56, 50, 102, 97, 102, 98, 45, 54, 98, 51, 49, 45, 52, 101, 98, 55, 45, 97, 99, 54, 50, 45, 52, 50, 53, 53, 97, 101, 100, 51, 99, 48, 56, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c." peer_addr = [fd00:1122:3344:101::5]:37909 req_id = 6979b477-164a-4661-8746-209627a2e5c3
1132026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), ), original: Query { name: Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 56, 57, 97, 51, 53, 98, 57, 45, 101, 98, 54, 101, 45, 52, 48, 56, 97, 45, 97, 48, 49, 97, 45, 101, 57, 56, 51, 57, 97, 98, 51, 52, 99, 97, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8." peer_addr = [fd00:1122:3344:101::4]:38101 req_id = c29d2a65-7666-4d13-8642-eca133f10793
1142026-07-29T16:45:00.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54786, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), ), original: Query { name: Name("oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 55, 56, 56, 50, 102, 97, 102, 98, 45, 54, 98, 51, 49, 45, 52, 101, 98, 55, 45, 97, 99, 54, 50, 45, 52, 50, 53, 53, 97, 101, 100, 51, 99, 48, 56, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_7882fafb-6b31-4eb7-ac62-4255aed3c08c." peer_addr = [fd00:1122:3344:101::5]:58005 req_id = dfe32124-f71a-4a21-ae2c-4bd8756f698a
1152026-07-29T16:45:00.088ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52745, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), ), original: Query { name: Name("oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 49, 56, 57, 97, 51, 53, 98, 57, 45, 101, 98, 54, 101, 45, 52, 48, 56, 97, 45, 97, 48, 49, 97, 45, 101, 57, 56, 51, 57, 97, 98, 51, 52, 99, 97, 56, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_189a35b9-eb6e-408a-a01a-e9839ab34ca8." peer_addr = [fd00:1122:3344:101::4]:63374 req_id = 6a06be2d-c55c-4bdd-878c-74170ee18a76
1162026-07-29T16:45:00.090ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), ), original: Query { name: Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 53, 98, 54, 52, 54, 48, 53, 45, 51, 57, 48, 49, 45, 52, 48, 100, 99, 45, 98, 48, 57, 48, 45, 56, 99, 49, 48, 56, 54, 56, 53, 57, 49, 102, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff." peer_addr = [fd00:1122:3344:101::7]:42676 req_id = b1bd8316-4efc-45bf-8329-e4ec050de202
1172026-07-29T16:45:00.090ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11942, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), ), original: Query { name: Name("oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 53, 98, 54, 52, 54, 48, 53, 45, 51, 57, 48, 49, 45, 52, 48, 100, 99, 45, 98, 48, 57, 48, 45, 56, 99, 49, 48, 56, 54, 56, 53, 57, 49, 102, 102, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_f5b64605-3901-40dc-b090-8c10868591ff." peer_addr = [fd00:1122:3344:101::7]:56960 req_id = d39543f9-a22c-4f75-8474-5a5164718e3b
1182026-07-29T16:45:00.633ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:51626 req_id = ceb2d788-d5d6-4a55-a729-a52346f9b283
1192026-07-29T16:45:00.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60622, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), ), original: Query { name: Name("oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 100, 98, 98, 48, 51, 57, 100, 45, 48, 101, 98, 56, 45, 52, 100, 52, 57, 45, 98, 49, 48, 56, 45, 51, 56, 102, 56, 51, 55, 100, 101, 48, 54, 54, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_6dbb039d-0eb8-4d49-b108-38f837de0669." peer_addr = [fd00:1122:3344:101::e]:36460 req_id = 29737a9f-c739-4c1a-a2b4-c28a92620263
1202026-07-29T16:45:01.013ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), ), original: Query { name: Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 102, 57, 53, 55, 101, 98, 101, 45, 53, 101, 52, 57, 45, 52, 57, 55, 55, 45, 97, 52, 50, 101, 45, 49, 53, 51, 52, 54, 52, 98, 51, 99, 56, 52, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847." peer_addr = [fd00:1122:3344:101::6]:33454 req_id = 6e4af753-bebf-4c76-bdbe-84d82b26b242
1212026-07-29T16:45:01.014ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13868, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), ), original: Query { name: Name("oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 102, 102, 57, 53, 55, 101, 98, 101, 45, 53, 101, 52, 57, 45, 52, 57, 55, 55, 45, 97, 52, 50, 101, 45, 49, 53, 51, 52, 54, 52, 98, 51, 99, 56, 52, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ff957ebe-5e49-4977-a42e-153464b3c847." peer_addr = [fd00:1122:3344:101::6]:41435 req_id = 091b3610-7dd3-4124-a8f1-ce8f88f7e3d1
1222026-07-29T16:45:01.110ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11174, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c."), ), original: Query { name: Name("oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 52, 53, 97, 53, 99, 48, 99, 97, 45, 57, 56, 98, 99, 45, 52, 48, 98, 52, 45, 56, 57, 50, 101, 45, 101, 51, 102, 53, 51, 98, 48, 56, 54, 98, 49, 99, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_45a5c0ca-98bc-40b4-892e-e3f53b086b1c." peer_addr = [fd00:1122:3344:101::1b]:58620 req_id = 51d56504-da07-41e5-909d-6d2653be5abf
1232026-07-29T16:45:01.281ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16154, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:63300 req_id = 89d22a6b-6acd-4dcd-9b5d-4fb705ab455a
1242026-07-29T16:45:01.281ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16154, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:38084 req_id = b86d6afd-955a-4bd0-a121-3054c7f33e14
1252026-07-29T16:45:01.379ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3157, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56899 req_id = 3081cf4d-0115-45cb-8f73-5ccc50f330b8