Max level shown:
1 [ Mar 24 15:24:55 Enabled. ]
2 [ Mar 24 15:24:55 Rereading configuration. ]
3 [ Mar 24 15:24:56 Rereading configuration. ]
4 [ Mar 24 15:24:57 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/dns-server/bin/dns-server --config-file /var/svc/manifest/site/internal_dns/config.toml --http-address [fd00:1122:3344:2::1]:5353 --dns-address [fd00:1122:3344:2::1]:53 &"). ]
5 [ Mar 24 15:24:57 Method "start" exited with status 0. ]
6 Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } }
7 note: configured to log to "/dev/stdout"
82026-03-24T15:24:57.598ZINFOdns-server: config config = Config { log: File { level: Info, path: "/dev/stdout", if_exists: Append }, dropshot: ConfigDropshot { bind_address: [fd00:1122:3344:2::1]:5353, default_request_body_max_bytes: 104857600, default_handler_task_mode: Detached, log_headers: [] }, storage: Config { storage_path: "/data/dns", keep_old_generations: 3 } } dns_config = Config { bind_address: [fd00:1122:3344:2::1]:53 }
92026-03-24T15:24:57.598ZINFOdns-server (store): opening sled database path = /data/dns
102026-03-24T15:24:57.618ZINFOdns-server (store): pruning trees for generations newer than 0
112026-03-24T15:24:57.618ZINFOdns-server (store): pruning trees for generations older than 0 keep = 3
122026-03-24T15:24:57.618ZINFOdns-server (dns): DNS server bound to address local_address = [fd00:1122:3344:2::1]:53
132026-03-24T15:24:57.619ZINFOdns-server (http): listening local_addr = [fd00:1122:3344:2::1]:5353
142026-03-24T15:24:58.583ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32279, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_nexus._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 6, 95, 110, 101, 120, 117, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "_nexus._tcp.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:2::2]:64908 req_id = 81ca8074-fa24-4967-a250-e1d0c1d7a2d3
152026-03-24T15:24:58.583ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1150, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("69fb25d2-69a8-4441-9088-29fdf431342b.host.control-plane.oxide.internal."), ), original: Query { name: Name("69fb25d2-69a8-4441-9088-29fdf431342b.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 54, 57, 102, 98, 50, 53, 100, 50, 45, 54, 57, 97, 56, 45, 52, 52, 52, 49, 45, 57, 48, 56, 56, 45, 50, 57, 102, 100, 102, 52, 51, 49, 51, 52, 50, 98, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "69fb25d2-69a8-4441-9088-29fdf431342b.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:2::2]:17774 req_id = 2d9515cd-eb12-4829-a0dc-146c6f1300ee
162026-03-24T15:24:58.584ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13582, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("22b10425-f533-450a-895d-4fa575e462a3.host.control-plane.oxide.internal."), ), original: Query { name: Name("22b10425-f533-450a-895d-4fa575e462a3.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 50, 50, 98, 49, 48, 52, 50, 53, 45, 102, 53, 51, 51, 45, 52, 53, 48, 97, 45, 56, 57, 53, 100, 45, 52, 102, 97, 53, 55, 53, 101, 52, 54, 50, 97, 51, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "22b10425-f533-450a-895d-4fa575e462a3.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:2::2]:12887 req_id = bb7acc57-2ac0-4420-b5b9-7900a8bc6f12
172026-03-24T15:24:58.584ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25750, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("9ef1d4f5-9c44-466c-8172-63f6f38adeb7.host.control-plane.oxide.internal."), ), original: Query { name: Name("9ef1d4f5-9c44-466c-8172-63f6f38adeb7.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN, }, }, ], original: [ 36, 57, 101, 102, 49, 100, 52, 102, 53, 45, 57, 99, 52, 52, 45, 52, 54, 54, 99, 45, 56, 49, 55, 50, 45, 54, 51, 102, 54, 102, 51, 56, 97, 100, 101, 98, 55, 4, 104, 111, 115, 116, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } SERVFAIL: server is not authoritative for name: "9ef1d4f5-9c44-466c-8172-63f6f38adeb7.host.control-plane.oxide.internal." peer_addr = [fd00:1122:3344:2::2]:2821 req_id = a0130842-b086-45e3-bc84-7394b8f6f52d
182026-03-24T15:24:58.600ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:2::2]:35884
192026-03-24T15:24:58.601ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = d0a5a0b1-a8dd-4ad8-80cd-9195a04ca707
202026-03-24T15:24:58.601ZINFOdns-server (store): pruning trees for generations newer than 0
212026-03-24T15:24:58.618ZINFOdns-server (store): pruning trees for generations older than 1 keep = 3
222026-03-24T15:24:58.618ZINFOdns-server (store): updated generation new_generation = 1 req_id = d0a5a0b1-a8dd-4ad8-80cd-9195a04ca707
232026-03-24T15:24:58.618ZINFOdns-server (http): request completed latency_us = 17787 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:2::2]:35884 req_id = d0a5a0b1-a8dd-4ad8-80cd-9195a04ca707 response_code = 204 uri = /config
242026-03-24T15:25:10.036ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15130, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:65329 req_id = dfafcdbe-a23f-4fdc-bee3-0cc331ca8088
252026-03-24T15:25:10.041ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36480, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:54501 req_id = 67759e3b-24cb-40e1-886b-6fe6ab0f422c
262026-03-24T15:25:10.045ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3429, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:49009 req_id = 9614e654-ed46-45a0-9e51-6055a41cf9aa
272026-03-24T15:26:49.325ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8773, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54501 req_id = 6236e193-7139-4aef-8624-3cf74d9b4fbc
282026-03-24T15:26:59.316ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 887, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51399 req_id = 397d139e-90bb-489e-a1cf-6b1a0b8d8755
292026-03-24T15:27:01.369ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12279, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:17806 req_id = 454abe54-3418-4e45-a856-5188e21b11d4
302026-03-24T15:27:01.447ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50142, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:63193 req_id = 04b94037-349f-4388-af4f-7c61775ba4ca
312026-03-24T15:27:01.479ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21835, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:41352 req_id = 3cff90d6-a440-4a81-9137-0eaca26a3329
322026-03-24T15:27:03.689ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34814, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), ), original: Query { name: Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 57, 55, 101, 102, 56, 102, 97, 51, 45, 99, 55, 51, 52, 45, 52, 51, 52, 54, 45, 97, 101, 54, 101, 45, 100, 101, 100, 57, 100, 57, 48, 53, 56, 50, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b." peer_addr = [fd00:1122:3344:101::e]:38684 req_id = 856c5910-5669-42cc-ad68-67871eecd158
332026-03-24T15:27:03.700ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34814, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), ), original: Query { name: Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 57, 55, 101, 102, 56, 102, 97, 51, 45, 99, 55, 51, 52, 45, 52, 51, 52, 54, 45, 97, 101, 54, 101, 45, 100, 101, 100, 57, 100, 57, 48, 53, 56, 50, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b." peer_addr = [fd00:1122:3344:101::e]:60113 req_id = edd2d313-10cc-4892-90d2-3132af867ed0
342026-03-24T15:27:09.318ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48289, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57869 req_id = 4624f41d-5876-4163-9a2b-ef97922cfdd5
352026-03-24T15:27:16.699ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55621, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:37402 req_id = dc10d36d-6885-45e8-bd4b-3a3a972bdb6f
362026-03-24T15:27:17.012ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:61476
372026-03-24T15:27:17.067ZINFOdns-server (http): request completed latency_us = 24879 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:61476 req_id = cf1578e7-0db1-44ac-a3ce-54163dfc0e16 response_code = 200 uri = /config
382026-03-24T15:27:18.140ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:39739
392026-03-24T15:27:18.143ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 061b6ed4-0d4c-4735-a7d7-0cab98d5143d
402026-03-24T15:27:18.143ZINFOdns-server (store): updated generation new_generation = 1 req_id = 061b6ed4-0d4c-4735-a7d7-0cab98d5143d
412026-03-24T15:27:18.143ZINFOdns-server (http): request completed latency_us = 2694 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:39739 req_id = 061b6ed4-0d4c-4735-a7d7-0cab98d5143d response_code = 204 uri = /config
422026-03-24T15:27:18.449ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14906, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:8037 req_id = 2871bbce-a1cb-40fa-9af5-41f0c8b1d296
432026-03-24T15:27:18.735ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:56343
442026-03-24T15:27:18.735ZINFOdns-server (http): request completed latency_us = 267 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:56343 req_id = f0f3c82f-caa3-4029-a2a4-a4ff0ad1eda3 response_code = 200 uri = /config
452026-03-24T15:27:19.320ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59013, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:51435 req_id = 79558e3b-afcd-4839-b239-e6b8b5ea526a
462026-03-24T15:27:19.706ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41491, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:47081 req_id = cb6453e0-3ffd-4398-aa61-fdd1efceddee
472026-03-24T15:27:19.926ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:46035
482026-03-24T15:27:19.927ZINFOdns-server (http): request completed latency_us = 228 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:46035 req_id = 3ece1f56-5deb-44f5-9c70-8dbe5e99fadc response_code = 200 uri = /config
492026-03-24T15:27:21.562ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9766, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:18280 req_id = dee64f7c-3202-41e4-b752-97bbb8826ed8
502026-03-24T15:27:21.594ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 593, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:33161 req_id = ad1114b1-fc56-4cdd-98b9-93cc0a196af2
512026-03-24T15:27:21.730ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:63313
522026-03-24T15:27:21.731ZINFOdns-server (http): request completed latency_us = 232 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:63313 req_id = 70027619-1af9-4704-9e93-a843325fd7b2 response_code = 200 uri = /config
532026-03-24T15:27:21.783ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:37489
542026-03-24T15:27:21.783ZINFOdns-server (http): request completed latency_us = 201 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:37489 req_id = 664d4e1e-5381-459f-887c-36b8bfa58c6e response_code = 200 uri = /config
552026-03-24T15:27:29.321ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8203, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:58882 req_id = 11bf8d9d-84e5-417f-b47c-4546a8afeaae
562026-03-24T15:27:39.322ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10096, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:50516 req_id = f7634c34-7f6c-4679-ab1b-694e4b2e8927
572026-03-24T15:27:49.323ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64410, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56416 req_id = d16dc5b6-9ab2-4088-b12c-d71d6473c600
582026-03-24T15:27:59.326ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47135, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49923 req_id = 3306d221-4178-4de4-a254-0fdf75fe1021
592026-03-24T15:28:00.881ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:45878
602026-03-24T15:28:00.882ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = ceabad69-fbdd-40b1-b895-672c94a77934
612026-03-24T15:28:00.882ZINFOdns-server (store): updated generation new_generation = 1 req_id = ceabad69-fbdd-40b1-b895-672c94a77934
622026-03-24T15:28:00.882ZINFOdns-server (http): request completed latency_us = 369 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:45878 req_id = ceabad69-fbdd-40b1-b895-672c94a77934 response_code = 204 uri = /config
632026-03-24T15:28:01.135ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:64891
642026-03-24T15:28:01.135ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 1590a056-59c9-4731-93ec-d80e3ccba595
652026-03-24T15:28:01.136ZINFOdns-server (store): updated generation new_generation = 1 req_id = 1590a056-59c9-4731-93ec-d80e3ccba595
662026-03-24T15:28:01.136ZINFOdns-server (http): request completed latency_us = 324 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:64891 req_id = 1590a056-59c9-4731-93ec-d80e3ccba595 response_code = 204 uri = /config
672026-03-24T15:28:01.203ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:33021
682026-03-24T15:28:01.204ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 3694164c-d7fd-463e-9162-faa807f07871
692026-03-24T15:28:01.204ZINFOdns-server (store): updated generation new_generation = 1 req_id = 3694164c-d7fd-463e-9162-faa807f07871
702026-03-24T15:28:01.204ZINFOdns-server (http): request completed latency_us = 329 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:33021 req_id = 3694164c-d7fd-463e-9162-faa807f07871 response_code = 204 uri = /config
712026-03-24T15:28:02.577ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36063, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:58927 req_id = 191d8576-3e58-4294-a4bd-3cc250460dc2
722026-03-24T15:28:02.579ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4342, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:50670 req_id = 65700b7d-f763-4214-ba41-95949ca17702
732026-03-24T15:28:02.623ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43149, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:33920 req_id = 42a74539-0f60-4445-a555-8e3722526131
742026-03-24T15:28:02.684ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:57535
752026-03-24T15:28:02.684ZINFOdns-server (http): request completed latency_us = 208 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:57535 req_id = e486beff-ad11-45c9-9d08-5a4a59ea6ae1 response_code = 200 uri = /config
762026-03-24T15:28:02.723ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:52960
772026-03-24T15:28:02.723ZINFOdns-server (http): request completed latency_us = 259 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:52960 req_id = 4fa1b577-64a6-4b9d-8bb4-dabcd9efc648 response_code = 200 uri = /config
782026-03-24T15:28:02.765ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:49041
792026-03-24T15:28:02.765ZINFOdns-server (http): request completed latency_us = 190 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:49041 req_id = db71f5b6-a2d0-466b-a3f4-7eeb73dff6f8 response_code = 200 uri = /config
802026-03-24T15:28:09.327ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1969, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56704 req_id = 49fe69bd-8152-4f37-8624-bb475ecb18d6
812026-03-24T15:28:19.329ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24068, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:49512 req_id = 0c127ccd-1856-4475-abb4-19b232d0f5db
822026-03-24T15:28:29.331ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28106, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:53646 req_id = eb257184-2e1c-497b-b74a-0b821bdb23fe
832026-03-24T15:28:39.333ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57726, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:56826 req_id = b5f4058a-588e-4ad8-a744-e20ec58ab7d8
842026-03-24T15:28:49.335ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40524, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:55408 req_id = 857d1fc9-96b1-4104-8aea-0a5ece43c24f
852026-03-24T15:28:59.336ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31419, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:59333 req_id = 65390aed-fe03-421b-85ce-07c6ed4ffea4
862026-03-24T15:29:00.874ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:54716
872026-03-24T15:29:00.875ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 40348939-b0f3-40e4-8810-944b7dd5ac13
882026-03-24T15:29:00.875ZINFOdns-server (store): updated generation new_generation = 1 req_id = 40348939-b0f3-40e4-8810-944b7dd5ac13
892026-03-24T15:29:00.875ZINFOdns-server (http): request completed latency_us = 277 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:54716 req_id = 40348939-b0f3-40e4-8810-944b7dd5ac13 response_code = 204 uri = /config
902026-03-24T15:29:00.882ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:59571
912026-03-24T15:29:00.882ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 5a137b50-9313-4887-bfdd-7802d7748530
922026-03-24T15:29:00.882ZINFOdns-server (store): updated generation new_generation = 1 req_id = 5a137b50-9313-4887-bfdd-7802d7748530
932026-03-24T15:29:00.882ZINFOdns-server (http): request completed latency_us = 193 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:59571 req_id = 5a137b50-9313-4887-bfdd-7802d7748530 response_code = 204 uri = /config
942026-03-24T15:29:00.908ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:44523
952026-03-24T15:29:00.908ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 60fa8d93-5a7f-474e-b51e-3960efb368e7
962026-03-24T15:29:00.908ZINFOdns-server (store): updated generation new_generation = 1 req_id = 60fa8d93-5a7f-474e-b51e-3960efb368e7
972026-03-24T15:29:00.908ZINFOdns-server (http): request completed latency_us = 396 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:44523 req_id = 60fa8d93-5a7f-474e-b51e-3960efb368e7 response_code = 204 uri = /config
982026-03-24T15:29:02.143ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64645, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:34069 req_id = f66e5bd2-9c79-49ac-81e0-eee321a9dc0c
992026-03-24T15:29:02.148ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53226, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:62118 req_id = d1a6ae63-ee5e-421e-a367-b558fa8e7c63
1002026-03-24T15:29:02.152ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 23256, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:53470 req_id = 08eb8fc5-f493-467b-8c62-135ef8529205
1012026-03-24T15:29:02.216ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34153, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::a]:33672 req_id = b6ef3787-11c8-4af3-ab6f-841dd15893aa
1022026-03-24T15:29:02.222ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 106, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:1260 req_id = 160d5e31-8639-49f3-8fc6-60ac20c1278b
1032026-03-24T15:29:02.317ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:39101
1042026-03-24T15:29:02.317ZINFOdns-server (http): request completed latency_us = 218 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::a]:39101 req_id = 1ce854f0-2450-4c7d-8e7d-e9dc3db87811 response_code = 200 uri = /config
1052026-03-24T15:29:02.353ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:61479
1062026-03-24T15:29:02.353ZINFOdns-server (http): request completed latency_us = 192 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:61479 req_id = 99a20412-19d6-44d0-aca8-ea3423deee70 response_code = 200 uri = /config
1072026-03-24T15:29:02.371ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44674, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:10041 req_id = 1e5183e7-1ea5-49aa-8856-a3f8f1088eee
1082026-03-24T15:29:02.463ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:65324
1092026-03-24T15:29:02.464ZINFOdns-server (http): request completed latency_us = 206 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:65324 req_id = 2da75320-f350-4fc4-ae8c-21871b185662 response_code = 200 uri = /config
1102026-03-24T15:29:09.337ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42461, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:65279 req_id = be2a6d40-e467-49f4-afbf-5113274c924c
1112026-03-24T15:29:12.103ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58447, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:45399 req_id = d1a7b429-9ced-4158-915c-9325baa7d03b
1122026-03-24T15:29:12.108ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 53420, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: AAAA, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 28, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:65180 req_id = cd7bb79f-cb62-4283-8e3d-48a8e8fc10e4
1132026-03-24T15:29:12.112ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48495, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("0.pool.ntp.org."), ), original: Query { name: Name("0.pool.ntp.org."), query_type: A, query_class: IN, }, }, ], original: [ 1, 48, 4, 112, 111, 111, 108, 3, 110, 116, 112, 3, 111, 114, 103, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "0.pool.ntp.org." peer_addr = [fd00:1122:3344:101::1b]:55564 req_id = 2f7160f9-622c-4785-985c-3c0227cd3d2d
1142026-03-24T15:29:19.338ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62780, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:57079 req_id = 705dccec-4527-4155-af1b-421419e633e1
1152026-03-24T15:29:29.339ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32249, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:63991 req_id = 884c191e-0822-43f3-a440-48c727bc1d46
1162026-03-24T15:29:39.340ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12839, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:62217 req_id = 801cbe0b-f2c7-4c07-b8dc-6054e7059309
1172026-03-24T15:29:49.342ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54493, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:54901 req_id = f6a31b44-21f4-42d7-8d75-93393d544a14
1182026-03-24T15:29:59.344ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14290, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-cluster-native._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 26, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 99, 108, 117, 115, 116, 101, 114, 45, 110, 97, 116, 105, 118, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::d]:64157 req_id = 300ad5c9-4bc1-4e28-98f0-f0954b487ee0
1192026-03-24T15:30:00.255ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38417, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), ), original: Query { name: Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 57, 55, 101, 102, 56, 102, 97, 51, 45, 99, 55, 51, 52, 45, 52, 51, 52, 54, 45, 97, 101, 54, 101, 45, 100, 101, 100, 57, 100, 57, 48, 53, 56, 50, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b." peer_addr = [fd00:1122:3344:101::e]:41944 req_id = 32accbc4-f691-4853-9268-4d658b627d67
1202026-03-24T15:30:00.256ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38417, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), ), original: Query { name: Name("oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b."), query_type: A, query_class: IN, }, }, ], original: [ 51, 111, 120, 122, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 95, 57, 55, 101, 102, 56, 102, 97, 51, 45, 99, 55, 51, 52, 45, 52, 51, 52, 54, 45, 97, 101, 54, 101, 45, 100, 101, 100, 57, 100, 57, 48, 53, 56, 50, 55, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_clickhouse_97ef8fa3-c734-4346-ae6e-ded9d905827b." peer_addr = [fd00:1122:3344:101::e]:35350 req_id = 60848d4b-b2a5-40b4-a857-4624b3bf50c2
1212026-03-24T15:30:00.441ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1803, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ba589882-146d-43b6-8128-f6bb0423cc2a."), ), original: Query { name: Name("oxz_cockroachdb_ba589882-146d-43b6-8128-f6bb0423cc2a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 97, 53, 56, 57, 56, 56, 50, 45, 49, 52, 54, 100, 45, 52, 51, 98, 54, 45, 56, 49, 50, 56, 45, 102, 54, 98, 98, 48, 52, 50, 51, 99, 99, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ba589882-146d-43b6-8128-f6bb0423cc2a." peer_addr = [fd00:1122:3344:101::7]:59108 req_id = 750bed52-8738-4963-9148-055a2ce23cd5
1222026-03-24T15:30:00.442ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1803, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ba589882-146d-43b6-8128-f6bb0423cc2a."), ), original: Query { name: Name("oxz_cockroachdb_ba589882-146d-43b6-8128-f6bb0423cc2a."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 97, 53, 56, 57, 56, 56, 50, 45, 49, 52, 54, 100, 45, 52, 51, 98, 54, 45, 56, 49, 50, 56, 45, 102, 54, 98, 98, 48, 52, 50, 51, 99, 99, 50, 97, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ba589882-146d-43b6-8128-f6bb0423cc2a." peer_addr = [fd00:1122:3344:101::7]:56363 req_id = fd53566d-157f-4dca-957e-6be2dca2072f
1232026-03-24T15:30:00.476ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31700, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b26196cc-bb93-4a1c-99e8-5003aca1cc22."), ), original: Query { name: Name("oxz_cockroachdb_b26196cc-bb93-4a1c-99e8-5003aca1cc22."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 50, 54, 49, 57, 54, 99, 99, 45, 98, 98, 57, 51, 45, 52, 97, 49, 99, 45, 57, 57, 101, 56, 45, 53, 48, 48, 51, 97, 99, 97, 49, 99, 99, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b26196cc-bb93-4a1c-99e8-5003aca1cc22." peer_addr = [fd00:1122:3344:101::3]:61868 req_id = 2a477f20-a95c-4c82-9e01-9b6a4fc2c4bb
1242026-03-24T15:30:00.477ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31700, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_b26196cc-bb93-4a1c-99e8-5003aca1cc22."), ), original: Query { name: Name("oxz_cockroachdb_b26196cc-bb93-4a1c-99e8-5003aca1cc22."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 50, 54, 49, 57, 54, 99, 99, 45, 98, 98, 57, 51, 45, 52, 97, 49, 99, 45, 57, 57, 101, 56, 45, 53, 48, 48, 51, 97, 99, 97, 49, 99, 99, 50, 50, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_b26196cc-bb93-4a1c-99e8-5003aca1cc22." peer_addr = [fd00:1122:3344:101::3]:59640 req_id = 21eba06d-448b-4942-bc9a-bac5db7f9931
1252026-03-24T15:30:00.583ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40419, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ba2edaff-6440-4039-befc-30a3f76b1565."), ), original: Query { name: Name("oxz_cockroachdb_ba2edaff-6440-4039-befc-30a3f76b1565."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 97, 50, 101, 100, 97, 102, 102, 45, 54, 52, 52, 48, 45, 52, 48, 51, 57, 45, 98, 101, 102, 99, 45, 51, 48, 97, 51, 102, 55, 54, 98, 49, 53, 54, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ba2edaff-6440-4039-befc-30a3f76b1565." peer_addr = [fd00:1122:3344:101::6]:40753 req_id = 8aaf7030-97ba-4b7c-8162-69ebcb6f8414
1262026-03-24T15:30:00.584ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40419, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_ba2edaff-6440-4039-befc-30a3f76b1565."), ), original: Query { name: Name("oxz_cockroachdb_ba2edaff-6440-4039-befc-30a3f76b1565."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 98, 97, 50, 101, 100, 97, 102, 102, 45, 54, 52, 52, 48, 45, 52, 48, 51, 57, 45, 98, 101, 102, 99, 45, 51, 48, 97, 51, 102, 55, 54, 98, 49, 53, 54, 53, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_ba2edaff-6440-4039-befc-30a3f76b1565." peer_addr = [fd00:1122:3344:101::6]:40715 req_id = 5382eaf8-803d-4f91-acbb-7431b040e750
1272026-03-24T15:30:00.636ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19553, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_cd3893b0-d9dd-4bdc-8c10-9e7db65dbfa4."), ), original: Query { name: Name("oxz_cockroachdb_cd3893b0-d9dd-4bdc-8c10-9e7db65dbfa4."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 100, 51, 56, 57, 51, 98, 48, 45, 100, 57, 100, 100, 45, 52, 98, 100, 99, 45, 56, 99, 49, 48, 45, 57, 101, 55, 100, 98, 54, 53, 100, 98, 102, 97, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_cd3893b0-d9dd-4bdc-8c10-9e7db65dbfa4." peer_addr = [fd00:1122:3344:101::4]:56978 req_id = f4ce3027-799c-462a-910e-f231190d3a6a
1282026-03-24T15:30:00.636ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19553, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_cd3893b0-d9dd-4bdc-8c10-9e7db65dbfa4."), ), original: Query { name: Name("oxz_cockroachdb_cd3893b0-d9dd-4bdc-8c10-9e7db65dbfa4."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 99, 100, 51, 56, 57, 51, 98, 48, 45, 100, 57, 100, 100, 45, 52, 98, 100, 99, 45, 56, 99, 49, 48, 45, 57, 101, 55, 100, 98, 54, 53, 100, 98, 102, 97, 52, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_cd3893b0-d9dd-4bdc-8c10-9e7db65dbfa4." peer_addr = [fd00:1122:3344:101::4]:51807 req_id = 4c9fce05-eacc-4a50-858a-7f4727907beb
1292026-03-24T15:30:00.645ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51319, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_41173991-a002-4a32-8db8-4cdbd1e2076d."), ), original: Query { name: Name("oxz_cockroachdb_41173991-a002-4a32-8db8-4cdbd1e2076d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 52, 49, 49, 55, 51, 57, 57, 49, 45, 97, 48, 48, 50, 45, 52, 97, 51, 50, 45, 56, 100, 98, 56, 45, 52, 99, 100, 98, 100, 49, 101, 50, 48, 55, 54, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_41173991-a002-4a32-8db8-4cdbd1e2076d." peer_addr = [fd00:1122:3344:101::5]:59310 req_id = 50add98a-e347-4de9-9389-8618f71622ab
1302026-03-24T15:30:00.645ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51319, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_cockroachdb_41173991-a002-4a32-8db8-4cdbd1e2076d."), ), original: Query { name: Name("oxz_cockroachdb_41173991-a002-4a32-8db8-4cdbd1e2076d."), query_type: A, query_class: IN, }, }, ], original: [ 52, 111, 120, 122, 95, 99, 111, 99, 107, 114, 111, 97, 99, 104, 100, 98, 95, 52, 49, 49, 55, 51, 57, 57, 49, 45, 97, 48, 48, 50, 45, 52, 97, 51, 50, 45, 56, 100, 98, 56, 45, 52, 99, 100, 98, 100, 49, 101, 50, 48, 55, 54, 100, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_cockroachdb_41173991-a002-4a32-8db8-4cdbd1e2076d." peer_addr = [fd00:1122:3344:101::5]:52448 req_id = fb30278e-0f62-4bbe-ad38-1b7217a5cb96
1312026-03-24T15:30:00.877ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:43453
1322026-03-24T15:30:00.877ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = 67b37920-689d-42da-ab8b-916e15a58eff
1332026-03-24T15:30:00.877ZINFOdns-server (store): updated generation new_generation = 1 req_id = 67b37920-689d-42da-ab8b-916e15a58eff
1342026-03-24T15:30:00.877ZINFOdns-server (http): request completed latency_us = 230 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::b]:43453 req_id = 67b37920-689d-42da-ab8b-916e15a58eff response_code = 204 uri = /config
1352026-03-24T15:30:00.881ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:57453
1362026-03-24T15:30:00.882ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = ea607871-528b-4a32-90dc-91464891881c
1372026-03-24T15:30:00.882ZINFOdns-server (store): updated generation new_generation = 1 req_id = ea607871-528b-4a32-90dc-91464891881c
1382026-03-24T15:30:00.882ZINFOdns-server (http): request completed latency_us = 351 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::c]:57453 req_id = ea607871-528b-4a32-90dc-91464891881c response_code = 204 uri = /config
1392026-03-24T15:30:00.908ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::a]:41462
1402026-03-24T15:30:00.909ZINFOdns-server (store): attempting generation update new_generation = 1 req_id = e6eae8b2-cd1c-4c2d-8277-bb89a0d18566
1412026-03-24T15:30:00.909ZINFOdns-server (store): updated generation new_generation = 1 req_id = e6eae8b2-cd1c-4c2d-8277-bb89a0d18566
1422026-03-24T15:30:00.909ZINFOdns-server (http): request completed latency_us = 423 local_addr = [fd00:1122:3344:2::1]:5353 method = PUT remote_addr = [fd00:1122:3344:101::a]:41462 req_id = e6eae8b2-cd1c-4c2d-8277-bb89a0d18566 response_code = 204 uri = /config
1432026-03-24T15:30:01.087ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27023, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_ntp_ab9a1c21-d4d2-43f1-8da7-049b0c75719b."), ), original: Query { name: Name("oxz_ntp_ab9a1c21-d4d2-43f1-8da7-049b0c75719b."), query_type: A, query_class: IN, }, }, ], original: [ 44, 111, 120, 122, 95, 110, 116, 112, 95, 97, 98, 57, 97, 49, 99, 50, 49, 45, 100, 52, 100, 50, 45, 52, 51, 102, 49, 45, 56, 100, 97, 55, 45, 48, 52, 57, 98, 48, 99, 55, 53, 55, 49, 57, 98, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_ntp_ab9a1c21-d4d2-43f1-8da7-049b0c75719b." peer_addr = [fd00:1122:3344:101::1b]:58287 req_id = 9af77ee2-7554-4a60-9934-b9c9a0466e3a
1442026-03-24T15:30:01.437ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55062, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:46446 req_id = 2d899d09-948a-451c-b6b2-dda73d395d95
1452026-03-24T15:30:01.438ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55062, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 0, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("oxz_switch."), ), original: Query { name: Name("oxz_switch."), query_type: A, query_class: IN, }, }, ], original: [ 10, 111, 120, 122, 95, 115, 119, 105, 116, 99, 104, 0, 0, 1, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: None, } SERVFAIL: server is not authoritative for name: "oxz_switch." peer_addr = [fd00:1122:3344:101::2]:37521 req_id = 1ba56d27-27a7-484e-8a77-0de42f2952f8
1462026-03-24T15:30:02.586ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3612, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::b]:30142 req_id = 556374e7-50d7-4f9e-a452-3cfd82b14045
1472026-03-24T15:30:02.594ZERROdns-server (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58590, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, queries: Queries { queries: [ LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, ], original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, flags: EdnsFlags { dnssec_ok: false, z: 0, }, max_payload: 1232, options: OPT { options: [], }, }, ), } NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal peer_addr = [fd00:1122:3344:101::c]:31715 req_id = 2451483c-ba22-4a23-bf9c-20a31ee0b347
1482026-03-24T15:30:02.706ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::c]:53107
1492026-03-24T15:30:02.706ZINFOdns-server (http): request completed latency_us = 219 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::c]:53107 req_id = bc405497-2136-4b35-a2a4-c5372d620f17 response_code = 200 uri = /config
1502026-03-24T15:30:02.796ZINFOdns-server (http): accepted connection local_addr = [fd00:1122:3344:2::1]:5353 remote_addr = [fd00:1122:3344:101::b]:56948
1512026-03-24T15:30:02.796ZINFOdns-server (http): request completed latency_us = 245 local_addr = [fd00:1122:3344:2::1]:5353 method = GET remote_addr = [fd00:1122:3344:101::b]:56948 req_id = c9e042b1-cb04-4bc1-8eca-80a39c025ab4 response_code = 200 uri = /config