Max level shown:
1 [ Jul 11 17:56:35 Enabled. ]
2 [ Jul 11 17:56:35 Rereading configuration. ]
3 [ Jul 11 17:56:36 Rereading configuration. ]
4 [ Jul 11 17:56:38 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:3::1]:5353 --dns-address [fd00:1122:3344:3::1]:53 &"). ]
5 [ Jul 11 17:56:38 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-07-11T17:56:38.188ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:3::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [], compression: None }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:3::1]:53 }
92026-07-11T17:56:38.188ZINFOdns-server (store): opening sled database path = /data/dns
102026-07-11T17:56:38.208ZINFOdns-server (store): pruning trees for generations newer than 0
112026-07-11T17:56:38.208ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-07-11T17:56:38.208ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:3::1]:53
132026-07-11T17:56:38.208ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:3::1]:5353
142026-07-11T17:56:38.340ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:3::2]:35934
152026-07-11T17:56:38.341ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = e1c01cd5-d2eb-4e91-b879-0d3991b97dda
162026-07-11T17:56:38.341ZINFOdns-server (store): pruning trees for generations newer than 0
172026-07-11T17:56:38.420ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
182026-07-11T17:56:38.420ZINFOdns-server (store): updated generation new_generation = 1 req_id = e1c01cd5-d2eb-4e91-b879-0d3991b97dda
192026-07-11T17:56:38.420ZINFOdns-server (http): request completed latency_us = 79426 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:3::2]:35934 req_id = e1c01cd5-d2eb-4e91-b879-0d3991b97dda response_code = 204 uri = /config
202026-07-11T17:56:51.696ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 770, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:44116 req_id = 28462002-32c6-4e5f-a26a-4f4067464662
212026-07-11T17:56:51.701ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14116, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:62892 req_id = 1ede2c61-9aae-4ada-99a3-d0b4633cec17
222026-07-11T17:56:51.705ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23629, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49521 req_id = 7e4f2466-0196-401e-bbb0-6391161bd704
232026-07-11T17:58:23.420ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59824 req_id = e58de1ba-7160-4792-a1d5-110b81956dce
242026-07-11T17:58:26.784ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63857, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:53189 req_id = db230be6-5512-478b-853e-e78939c3156f
252026-07-11T17:58:26.785ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63857, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:47666 req_id = be1d3041-fc2c-43be-8464-7cfa6c872b9a
262026-07-11T17:58:31.568ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48831, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:28042 req_id = 0c6a57b7-0277-4415-ba62-cb7c94ce5287
272026-07-11T17:58:32.801ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25043, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:30086 req_id = 1f0baef6-f60e-48a9-8c9b-cfb5c3797f2a
282026-07-11T17:58:32.825ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35870, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:4818 req_id = 3eb10226-dc3d-49a6-af8b-4de770e34513
292026-07-11T17:58:33.400ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16702, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58213 req_id = 44b73e1b-bfb8-47b6-93ad-655d05037732
302026-07-11T17:58:43.402ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25912, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63946 req_id = 1d47f112-ff3c-4d1b-933f-733b84640f11
312026-07-11T17:58:53.404ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40386, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:61144 req_id = 74d2084f-66f4-49e3-a969-1c15f201f7d4
322026-07-11T17:58:55.634ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36121, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:11294 req_id = 6b7e1d3c-e43b-4bd8-a5f5-8dcb6c467f0f
332026-07-11T17:58:55.871ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:59229
342026-07-11T17:58:55.938ZINFOdns-server (http): request completed latency_us = 42369 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:59229 req_id = 0219d60f-6574-4668-86cc-ed24693aeea1 response_code = 200 uri = /config
352026-07-11T17:59:00.465ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38043, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:45756 req_id = f86723cc-f2e3-47c5-881e-24aa0bcefac9
362026-07-11T17:59:00.483ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:51186
372026-07-11T17:59:00.489ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 542a1173-7ecb-4daa-9845-d49416cd448a
382026-07-11T17:59:00.489ZINFOdns-server (store): updated generation new_generation = 1 req_id = 542a1173-7ecb-4daa-9845-d49416cd448a
392026-07-11T17:59:00.489ZINFOdns-server (http): request completed latency_us = 2782 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:51186 req_id = 542a1173-7ecb-4daa-9845-d49416cd448a response_code = 204 uri = /config
402026-07-11T17:59:00.760ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:57520
412026-07-11T17:59:00.761ZINFOdns-server (http): request completed latency_us = 301 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:57520 req_id = a82ab3d3-25a0-44a8-9582-b5e9354c0939 response_code = 200 uri = /config
422026-07-11T17:59:01.661ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26327, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:22409 req_id = 85868e9d-930f-4494-9e53-cd8afae9f7be
432026-07-11T17:59:01.904ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:39788
442026-07-11T17:59:01.904ZINFOdns-server (http): request completed latency_us = 216 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:39788 req_id = 41f34678-17a6-4794-a61a-6f2836469bdc response_code = 200 uri = /config
452026-07-11T17:59:02.387ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 29550, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:20326 req_id = e3a817d7-9ffc-45ea-98f6-f5ed811e83e2
462026-07-11T17:59:02.519ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:44138
472026-07-11T17:59:02.519ZINFOdns-server (http): request completed latency_us = 324 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:44138 req_id = 16f43bf0-7076-4fbf-bfa5-7ac263562c97 response_code = 200 uri = /config
482026-07-11T17:59:03.406ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37583, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58454 req_id = 73caa742-fb68-475e-aac6-b5c34d17a880
492026-07-11T17:59:03.752ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40865, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:55272 req_id = 4ea7be88-9aed-4b8b-aed6-8211e5730c05
502026-07-11T17:59:03.880ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:47275
512026-07-11T17:59:03.881ZINFOdns-server (http): request completed latency_us = 225 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:47275 req_id = d86818cb-7277-42fb-b155-7156b6717d71 response_code = 200 uri = /config
522026-07-11T17:59:13.408ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5499, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50015 req_id = c161e93f-456e-4d80-9007-cd558054353d
532026-07-11T17:59:23.409ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22583, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:52553 req_id = 6de978f7-b018-471b-accc-273c20c4a835
542026-07-11T17:59:31.483ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:38028
552026-07-11T17:59:31.485ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 57a0165d-48a4-43a9-9488-2c191965e992
562026-07-11T17:59:31.485ZINFOdns-server (store): updated generation new_generation = 1 req_id = 57a0165d-48a4-43a9-9488-2c191965e992
572026-07-11T17:59:31.485ZINFOdns-server (http): request completed latency_us = 246 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:38028 req_id = 57a0165d-48a4-43a9-9488-2c191965e992 response_code = 204 uri = /config
582026-07-11T17:59:32.520ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:40718
592026-07-11T17:59:32.523ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = b17d60b1-c3a8-4e52-82ee-a13b0648520e
602026-07-11T17:59:32.523ZINFOdns-server (store): updated generation new_generation = 1 req_id = b17d60b1-c3a8-4e52-82ee-a13b0648520e
612026-07-11T17:59:32.523ZINFOdns-server (http): request completed latency_us = 286 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:40718 req_id = b17d60b1-c3a8-4e52-82ee-a13b0648520e response_code = 204 uri = /config
622026-07-11T17:59:32.826ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:38590
632026-07-11T17:59:32.829ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 4b844b3a-691a-46fc-afbc-d54974964af6
642026-07-11T17:59:32.829ZINFOdns-server (store): updated generation new_generation = 1 req_id = 4b844b3a-691a-46fc-afbc-d54974964af6
652026-07-11T17:59:32.829ZINFOdns-server (http): request completed latency_us = 370 local_addr = [fd00:1122:3344:3::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:38590 req_id = 4b844b3a-691a-46fc-afbc-d54974964af6 response_code = 204 uri = /config
662026-07-11T17:59:33.193ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12983, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:21214 req_id = 9b36ea74-2609-4fd2-9a49-d5ec08b28f15
672026-07-11T17:59:33.296ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::b]:52603
682026-07-11T17:59:33.296ZINFOdns-server (http): request completed latency_us = 192 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:52603 req_id = 5df3e2e4-d8d5-411d-91c4-f690c2870685 response_code = 200 uri = /config
692026-07-11T17:59:33.411ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44046, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64025 req_id = bcf47f10-00e4-4338-9e31-4e62c97cf54b
702026-07-11T17:59:33.778ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39493, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:55681 req_id = 825437dd-3fe4-43e5-88cc-e9a5e0c0376c
712026-07-11T17:59:33.839ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25296, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:26783 req_id = 75e61a12-8647-4948-b3b1-1216df0955b4
722026-07-11T17:59:33.944ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::a]:53676
732026-07-11T17:59:33.944ZINFOdns-server (http): request completed latency_us = 199 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:53676 req_id = 52ebb21d-dda2-47a2-b741-c7f05f9b3cc5 response_code = 200 uri = /config
742026-07-11T17:59:33.985ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:3::1]:5353 remote_addr = [fd00:1122:3344:101::c]:61094
752026-07-11T17:59:33.985ZINFOdns-server (http): request completed latency_us = 177 local_addr = [fd00:1122:3344:3::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:61094 req_id = d9be6fa2-cac5-4486-a379-67df8366e176 response_code = 200 uri = /config
762026-07-11T17:59:43.413ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61276, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58328 req_id = b97862e7-af90-4c98-bb78-f16a918b0264
772026-07-11T17:59:53.414ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8126, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59789 req_id = aec25616-5086-49a2-a8ac-03891a4e0899
782026-07-11T18:00:00.100ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:37190 req_id = 11255542-7a51-4f20-a7fa-8ef17511cb6e
792026-07-11T18:00:00.100ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 35914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), ), original: Query { name: Name("oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 54, 53, 97, 52, 48, 50, 97, 50, 45, 99, 51, 52, 97, 45, 52, 97, 51, 101, 45, 97, 97, 98, 51, 45, 99, 101, 99, 98, 48, 100, 56, 53, 53, 51, 101, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_65a402a2-c34a-4a3e-aab3-cecb0d8553e9." peer_addr = [fd00:1122:3344:101::e]:49426 req_id = 62f71dbe-3b66-434b-a9ca-7cfb13c3d39f
802026-07-11T18:00:00.486ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14642, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_cf56f114-d8ea-4de5-9868-dbbf52c8d8e0."), ), original: Query { name: Name("oxz_ntp_cf56f114-d8ea-4de5-9868-dbbf52c8d8e0."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 99, 102, 53, 54, 102, 49, 49, 52, 45, 100, 56, 101, 97, 45, 52, 100, 101, 53, 45, 57, 56, 54, 56, 45, 100, 98, 98, 102, 53, 50, 99, 56, 100, 56, 101, 48, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_cf56f114-d8ea-4de5-9868-dbbf52c8d8e0." peer_addr = [fd00:1122:3344:101::1b]:47887 req_id = 9726f31d-6f4a-40be-b098-6057c4ab4222
812026-07-11T18:00:00.661ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60549, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:40491 req_id = 8b5718c9-a331-4c95-8065-9208706631c3
822026-07-11T18:00:00.661ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60549, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:33554 req_id = f9ae991b-d824-413e-afe1-0cfd337cdd3e
832026-07-11T18:00:00.805ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19565, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), ), original: Query { name: Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 57, 99, 56, 50, 54, 53, 55, 45, 101, 53, 101, 51, 45, 52, 100, 57, 100, 45, 56, 51, 99, 51, 45, 102, 55, 52, 101, 48, 98, 50, 52, 48, 102, 50, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24." peer_addr = [fd00:1122:3344:101::6]:32971 req_id = b77c275d-e406-4987-9ae3-fff551af681b
842026-07-11T18:00:00.806ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19565, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), ), original: Query { name: Name("oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 57, 99, 56, 50, 54, 53, 55, 45, 101, 53, 101, 51, 45, 52, 100, 57, 100, 45, 56, 51, 99, 51, 45, 102, 55, 52, 101, 48, 98, 50, 52, 48, 102, 50, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b9c82657-e5e3-4d9d-83c3-f74e0b240f24." peer_addr = [fd00:1122:3344:101::6]:48585 req_id = 61018288-1d01-4674-af4b-c68c3b04d00c
852026-07-11T18:00:00.832ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65052, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), ), original: Query { name: Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 52, 50, 49, 55, 97, 50, 98, 45, 54, 97, 99, 55, 45, 52, 98, 100, 100, 45, 97, 97, 55, 56, 45, 101, 53, 49, 52, 54, 100, 56, 51, 99, 57, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949." peer_addr = [fd00:1122:3344:101::5]:35736 req_id = 610233fa-9997-47ef-a52d-c7082bc76920
862026-07-11T18:00:00.833ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65052, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), ), original: Query { name: Name("oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 100, 52, 50, 49, 55, 97, 50, 98, 45, 54, 97, 99, 55, 45, 52, 98, 100, 100, 45, 97, 97, 55, 56, 45, 101, 53, 49, 52, 54, 100, 56, 51, 99, 57, 52, 57, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_d4217a2b-6ac7-4bdd-aa78-e5146d83c949." peer_addr = [fd00:1122:3344:101::5]:59529 req_id = c0ac77e3-62f5-4778-83f1-04e62d52008e
872026-07-11T18:00:00.891ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8485, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), ), original: Query { name: Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 100, 100, 52, 49, 57, 101, 49, 45, 97, 98, 100, 57, 45, 52, 52, 49, 52, 45, 56, 50, 51, 101, 45, 49, 54, 50, 56, 102, 56, 98, 98, 51, 49, 54, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167." peer_addr = [fd00:1122:3344:101::7]:63429 req_id = 5bcf176a-7cab-44f1-af97-e9a8de9a8667
882026-07-11T18:00:00.892ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8485, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), ), original: Query { name: Name("oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 50, 100, 100, 52, 49, 57, 101, 49, 45, 97, 98, 100, 57, 45, 52, 52, 49, 52, 45, 56, 50, 51, 101, 45, 49, 54, 50, 56, 102, 56, 98, 98, 51, 49, 54, 55, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_2dd419e1-abd9-4414-823e-1628f8bb3167." peer_addr = [fd00:1122:3344:101::7]:46121 req_id = 90f85a32-659f-476f-aafc-9ea9dd2e054d
892026-07-11T18:00:00.965ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), ), original: Query { name: Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 54, 56, 50, 54, 53, 98, 51, 45, 97, 50, 100, 99, 45, 52, 98, 54, 102, 45, 97, 49, 99, 101, 45, 54, 55, 97, 102, 100, 54, 57, 50, 97, 49, 97, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5." peer_addr = [fd00:1122:3344:101::3]:35228 req_id = 475513de-d8d7-485b-a8ab-e40a6bf4e60f
902026-07-11T18:00:00.966ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59685, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), ), original: Query { name: Name("oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 54, 54, 56, 50, 54, 53, 98, 51, 45, 97, 50, 100, 99, 45, 52, 98, 54, 102, 45, 97, 49, 99, 101, 45, 54, 55, 97, 102, 100, 54, 57, 50, 97, 49, 97, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_668265b3-a2dc-4b6f-a1ce-67afd692a1a5." peer_addr = [fd00:1122:3344:101::3]:35624 req_id = 54b47bf0-2b43-45b3-9d88-00521caec1f3
912026-07-11T18:00:00.969ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22323, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), ), original: Query { name: Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 57, 57, 50, 97, 101, 53, 54, 45, 99, 53, 54, 98, 45, 52, 98, 102, 55, 45, 56, 101, 56, 49, 45, 100, 101, 98, 52, 98, 97, 101, 99, 56, 98, 51, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d." peer_addr = [fd00:1122:3344:101::4]:44970 req_id = 9bff9056-6a35-4847-bd6d-1f7e57549674
922026-07-11T18:00:00.969ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22323, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), ), original: Query { name: Name("oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 51, 57, 57, 50, 97, 101, 53, 54, 45, 99, 53, 54, 98, 45, 52, 98, 102, 55, 45, 56, 101, 56, 49, 45, 100, 101, 98, 52, 98, 97, 101, 99, 56, 98, 51, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_3992ae56-c56b-4bf7-8e81-deb4baec8b3d." peer_addr = [fd00:1122:3344:101::4]:62222 req_id = e963dfd6-ed4e-4eed-ab10-dbfb80dab34a