{"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 21351,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"oxz_switch.\"),\n ),\n original: Query {\n name: Name(\"oxz_switch.\"),\n query_type: A,\n query_class: IN,\n },\n },\n ],\n original: [\n 10,\n 111,\n 120,\n 122,\n 95,\n 115,\n 119,\n 105,\n 116,\n 99,\n 104,\n 0,\n 0,\n 1,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"oxz_switch.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:00.582416795Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::2]:43686","req_id":"e7aaa475-87e3-40ee-a53c-4828069b5de8","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 21351,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"oxz_switch.\"),\n ),\n original: Query {\n name: Name(\"oxz_switch.\"),\n query_type: A,\n query_class: IN,\n },\n },\n ],\n original: [\n 10,\n 111,\n 120,\n 122,\n 95,\n 115,\n 119,\n 105,\n 116,\n 99,\n 104,\n 0,\n 0,\n 1,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"oxz_switch.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:00.583018237Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::2]:42222","req_id":"dc81fe2e-32cf-4333-a7c2-f27648e19620","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 47725,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"oxz_ntp_c1bc3a38-5cfd-409b-92e0-8e0fa6bcbbfd.\"),\n ),\n original: Query {\n name: Name(\"oxz_ntp_c1bc3a38-5cfd-409b-92e0-8e0fa6bcbbfd.\"),\n query_type: A,\n query_class: IN,\n },\n },\n ],\n original: [\n 44,\n 111,\n 120,\n 122,\n 95,\n 110,\n 116,\n 112,\n 95,\n 99,\n 49,\n 98,\n 99,\n 51,\n 97,\n 51,\n 56,\n 45,\n 53,\n 99,\n 102,\n 100,\n 45,\n 52,\n 48,\n 57,\n 98,\n 45,\n 57,\n 50,\n 101,\n 48,\n 45,\n 56,\n 101,\n 48,\n 102,\n 97,\n 54,\n 98,\n 99,\n 98,\n 98,\n 102,\n 100,\n 0,\n 0,\n 1,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"oxz_ntp_c1bc3a38-5cfd-409b-92e0-8e0fa6bcbbfd.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:00.933709908Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::1b]:37123","req_id":"dc302267-5e15-4175-bde8-e2ba1d5a4e65","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 8456,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"oxz_clickhouse_e7ac5100-6907-4886-802f-23e4a6e51faf.\"),\n ),\n original: Query {\n name: Name(\"oxz_clickhouse_e7ac5100-6907-4886-802f-23e4a6e51faf.\"),\n query_type: A,\n query_class: IN,\n },\n },\n ],\n original: [\n 51,\n 111,\n 120,\n 122,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 95,\n 101,\n 55,\n 97,\n 99,\n 53,\n 49,\n 48,\n 48,\n 45,\n 54,\n 57,\n 48,\n 55,\n 45,\n 52,\n 56,\n 56,\n 54,\n 45,\n 56,\n 48,\n 50,\n 102,\n 45,\n 50,\n 51,\n 101,\n 52,\n 97,\n 54,\n 101,\n 53,\n 49,\n 102,\n 97,\n 102,\n 0,\n 0,\n 1,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"oxz_clickhouse_e7ac5100-6907-4886-802f-23e4a6e51faf.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:01.004518002Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::e]:35647","req_id":"9ff3d969-28ce-47f9-8223-0130ae535d2d","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 8456,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"oxz_clickhouse_e7ac5100-6907-4886-802f-23e4a6e51faf.\"),\n ),\n original: Query {\n name: Name(\"oxz_clickhouse_e7ac5100-6907-4886-802f-23e4a6e51faf.\"),\n query_type: A,\n query_class: IN,\n },\n },\n ],\n original: [\n 51,\n 111,\n 120,\n 122,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 95,\n 101,\n 55,\n 97,\n 99,\n 53,\n 49,\n 48,\n 48,\n 45,\n 54,\n 57,\n 48,\n 55,\n 45,\n 52,\n 56,\n 56,\n 54,\n 45,\n 56,\n 48,\n 50,\n 102,\n 45,\n 50,\n 51,\n 101,\n 52,\n 97,\n 54,\n 101,\n 53,\n 49,\n 102,\n 97,\n 102,\n 0,\n 0,\n 1,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"oxz_clickhouse_e7ac5100-6907-4886-802f-23e4a6e51faf.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:01.005093244Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::e]:44228","req_id":"9ad42e3a-1827-46d4-ab59-bb127f5c66b5","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 29052,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:02.371185748Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:61246","req_id":"3024b6f2-5bc6-4834-8077-c98cd4b27f1e","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 17159,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:12.372724039Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:54547","req_id":"07fc5817-4985-4310-9ef4-d3202ba324a9","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:12.942587829Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:52293"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:12.944269563Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"53fdcc29-8b7f-4ae4-85b6-132c79833bcc","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:12.944305344Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"53fdcc29-8b7f-4ae4-85b6-132c79833bcc","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:12.944317074Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"53fdcc29-8b7f-4ae4-85b6-132c79833bcc","remote_addr":"[fd00:1122:3344:101::c]:52293","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":217,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 34192,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:13.684084138Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::c]:64196","req_id":"c07dd765-b60e-48f2-a389-1d333a696c45","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:13.779704798Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:43135"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:13.779964883Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"1704d36a-564c-4050-a5db-7d1a9fdef3ad","remote_addr":"[fd00:1122:3344:101::c]:43135","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":220,"response_code":200} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:14.193019295Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:45864"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:14.19427287Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"6fbb6045-3d54-44df-8e7c-6b3380fbfeee","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:14.194309551Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"6fbb6045-3d54-44df-8e7c-6b3380fbfeee","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:14.194320451Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"6fbb6045-3d54-44df-8e7c-6b3380fbfeee","remote_addr":"[fd00:1122:3344:101::b]:45864","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":238,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 61551,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:14.829450032Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::b]:32454","req_id":"ce044ac4-ab50-485c-9e3e-72491abe6e52","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:14.926870488Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:63924"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:14.927084193Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"5ea4b6ff-4706-4e9e-b0a6-49553f932277","remote_addr":"[fd00:1122:3344:101::b]:63924","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":183,"response_code":200} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:21.580853003Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::a]:59219"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:21.582130409Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"acdf9803-c0b4-4c10-b310-03d94e43d379","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:21.582170459Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"acdf9803-c0b4-4c10-b310-03d94e43d379","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:21.58218254Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"acdf9803-c0b4-4c10-b310-03d94e43d379","remote_addr":"[fd00:1122:3344:101::a]:59219","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":238,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 60107,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:22.27028586Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::a]:47887","req_id":"13d7d60e-9cd9-4749-9ef0-3222847b012c","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:22.368816339Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::a]:44266"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:15:22.369055674Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"fa9259e3-ebd3-4f0d-807a-ba73b220ce49","remote_addr":"[fd00:1122:3344:101::a]:44266","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":190,"response_code":200} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 32998,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:22.374459004Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:64027","req_id":"38b7c3e0-88cc-416c-b106-73e660b2bdb2","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 9853,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:32.375991455Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:54498","req_id":"8f449585-22c4-4a95-a435-4509f8b9e232","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 56004,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:42.378117948Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:60005","req_id":"2beb7520-46af-4bae-80b8-a1bf8ae2aeb5","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 41450,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:15:52.379890084Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:64233","req_id":"cf859b6b-2921-4958-93c6-4625328a194a","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 32372,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:02.381641559Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:63907","req_id":"546a5b96-10fe-4cff-b5f4-ae9c474e1e7f","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 29506,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:12.383698421Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:60887","req_id":"2310b545-20b0-4b83-8400-a4a4b021f873","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:12.942595087Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:35928"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:12.94419695Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"8b9edcd5-5f42-47b0-a25e-340b688f5493","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:12.94423092Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"8b9edcd5-5f42-47b0-a25e-340b688f5493","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:12.944244081Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"8b9edcd5-5f42-47b0-a25e-340b688f5493","remote_addr":"[fd00:1122:3344:101::c]:35928","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":269,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 26571,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:13.895341163Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::c]:50702","req_id":"a97abae9-dbe5-42ae-a624-8167c74f8f05","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:13.992479924Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:38812"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:13.992700019Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"d3b1e0c0-f0f8-4e8b-abeb-72b83467c22e","remote_addr":"[fd00:1122:3344:101::c]:38812","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":185,"response_code":200} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:14.192526903Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:45142"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:14.19387572Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"d41b53ba-ab47-463a-bd7d-39ce81996571","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:14.193911161Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"d41b53ba-ab47-463a-bd7d-39ce81996571","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:14.193919731Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"d41b53ba-ab47-463a-bd7d-39ce81996571","remote_addr":"[fd00:1122:3344:101::b]:45142","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":198,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 37511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:15.040678436Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::b]:30589","req_id":"728eb94d-623a-4781-9807-b5a584da73aa","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:15.145732169Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:42777"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:15.145985634Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"cde7459a-fccc-4f73-8bcb-677486a4bc26","remote_addr":"[fd00:1122:3344:101::b]:42777","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":217,"response_code":200} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:21.581006874Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::a]:51431"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:21.582214218Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"2dae7bb6-47c2-4b55-b900-8b31430c520b","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:21.582249949Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"2dae7bb6-47c2-4b55-b900-8b31430c520b","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:21.58226207Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"2dae7bb6-47c2-4b55-b900-8b31430c520b","remote_addr":"[fd00:1122:3344:101::a]:51431","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":246,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 63754,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:22.345852119Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::a]:50781","req_id":"9c884adf-7c4f-425c-884b-b978661ed01b","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 5050,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:22.38515811Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:57744","req_id":"41850497-895f-43f5-b13e-2eeb3c2d5c83","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:22.445047491Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::a]:59520"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:16:22.445315077Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"a19a330b-2e99-4417-a742-a2ea9dd652fe","remote_addr":"[fd00:1122:3344:101::a]:59520","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":215,"response_code":200} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 21476,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:32.386897056Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:62762","req_id":"a3f3a3cf-8714-4a04-8370-ddd63a1da5e2","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 58372,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:42.389257593Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:55043","req_id":"94faa699-62b6-475b-b5f5-123acce1703f","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 6606,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:52.39057624Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:57799","req_id":"f1afa55b-61ae-4365-a072-8403e190038d","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 23773,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"0.pool.ntp.org.\"),\n ),\n original: Query {\n name: Name(\"0.pool.ntp.org.\"),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 1,\n 48,\n 4,\n 112,\n 111,\n 111,\n 108,\n 3,\n 110,\n 116,\n 112,\n 3,\n 111,\n 114,\n 103,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"0.pool.ntp.org.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:55.049825512Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::1b]:49869","req_id":"470e89b2-bc22-4684-936c-6cab421b6f95","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 8037,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"0.pool.ntp.org.\"),\n ),\n original: Query {\n name: Name(\"0.pool.ntp.org.\"),\n query_type: AAAA,\n query_class: IN,\n },\n },\n ],\n original: [\n 1,\n 48,\n 4,\n 112,\n 111,\n 111,\n 108,\n 3,\n 110,\n 116,\n 112,\n 3,\n 111,\n 114,\n 103,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"0.pool.ntp.org.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:55.061779416Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::1b]:43783","req_id":"47a3a55d-08c5-4055-91ad-e54302c68269","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 4386,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 0,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"0.pool.ntp.org.\"),\n ),\n original: Query {\n name: Name(\"0.pool.ntp.org.\"),\n query_type: A,\n query_class: IN,\n },\n },\n ],\n original: [\n 1,\n 48,\n 4,\n 112,\n 111,\n 111,\n 108,\n 3,\n 110,\n 116,\n 112,\n 3,\n 111,\n 114,\n 103,\n 0,\n 0,\n 1,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: None,\n} SERVFAIL: server is not authoritative for name: \"0.pool.ntp.org.\"","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:16:55.07719397Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::1b]:33562","req_id":"b8c6e93c-b0f6-4cf1-a280-41c8adb68fd6","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 41222,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:02.392210573Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:55055","req_id":"a0d252bc-3654-4caa-919a-13fcab7265ff","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 5451,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:12.393668634Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:55572","req_id":"8dfd8a3d-1ebe-4bcb-9683-f86ec4835fe9","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:12.943813932Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:47938"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:12.945316633Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"78637fde-3c6c-4cb7-98b2-e408ead10e4b","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:12.945375264Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"78637fde-3c6c-4cb7-98b2-e408ead10e4b","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:12.945392304Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"78637fde-3c6c-4cb7-98b2-e408ead10e4b","remote_addr":"[fd00:1122:3344:101::c]:47938","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":164,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 60386,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:13.717587479Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::c]:64231","req_id":"a24493f7-de34-4ef5-aefa-f5170cd2c96f","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:13.818115089Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:42648"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:13.818327554Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"5fde1f91-fb1b-46c3-a08e-bebe46b39d84","remote_addr":"[fd00:1122:3344:101::c]:42648","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":180,"response_code":200} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:14.19334753Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:37230"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:14.194683427Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"07f020c8-0e07-4d14-ad82-66b1dfba969b","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:14.194718558Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"07f020c8-0e07-4d14-ad82-66b1dfba969b","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:14.194728479Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"07f020c8-0e07-4d14-ad82-66b1dfba969b","remote_addr":"[fd00:1122:3344:101::b]:37230","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":206,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 57105,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:14.87600925Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::b]:2233","req_id":"ec789cc6-406c-4f66-b863-1ca0262e186a","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:15.065133386Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:54875"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:15.065383051Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"bb91d190-5561-4b78-8f82-28d97951f708","remote_addr":"[fd00:1122:3344:101::b]:54875","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":209,"response_code":200} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:21.58214536Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::a]:43974"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:21.583468737Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"feff9157-f971-4bdc-9264-39188de7581c","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:21.583501028Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"feff9157-f971-4bdc-9264-39188de7581c","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:21.583508678Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"feff9157-f971-4bdc-9264-39188de7581c","remote_addr":"[fd00:1122:3344:101::a]:43974","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":190,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 53070,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:22.395483324Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:58204","req_id":"2241dae1-f190-46d7-9f48-8c3b951c08dd","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 14381,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:22.686679172Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::a]:7767","req_id":"fb85ded9-c588-4bdf-96f0-1c881811042a","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:22.784085538Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::a]:54605"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:17:22.784333323Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"550fbfbd-782a-4678-9cd3-867720707c0c","remote_addr":"[fd00:1122:3344:101::a]:54605","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":207,"response_code":200} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 28499,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:32.396349834Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:50111","req_id":"13b8eea1-9743-4789-b0a3-c48d5173884c","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 33578,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:42.398784654Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:49473","req_id":"ff0fd883-32e3-48ec-b553-0b7c086c7ebc","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 1841,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:17:52.400563911Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:58406","req_id":"89107fc5-8c55-4c8c-be65-098c44bf67d7","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 26140,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:18:02.402865818Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:51629","req_id":"220abfd0-7324-4506-8050-225ac60ea627","component":"dns"} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 45638,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-cluster-native._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 26,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 99,\n 108,\n 117,\n 115,\n 116,\n 101,\n 114,\n 45,\n 110,\n 97,\n 116,\n 105,\n 118,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-cluster-native._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:18:12.405088683Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::d]:49470","req_id":"a9c96efa-16e4-49f8-aa32-eb4c4dd1a13e","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:12.941749326Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:63244"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:12.943280557Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"5c149c3e-8594-4b2b-8fd4-d3ffdd284587","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:12.943314358Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"5c149c3e-8594-4b2b-8fd4-d3ffdd284587","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:12.943325568Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"5c149c3e-8594-4b2b-8fd4-d3ffdd284587","remote_addr":"[fd00:1122:3344:101::c]:63244","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":216,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 15488,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:18:13.855995378Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::c]:21419","req_id":"a72edc3c-97c1-4222-91fb-85f3a457e285","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:14.031001086Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::c]:53716"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:14.031246211Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"ebc6793d-771d-460a-a6db-fdc67a93b95a","remote_addr":"[fd00:1122:3344:101::c]:53716","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":211,"response_code":200} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:14.192865616Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:37639"} {"msg":"attempting generation update","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:14.194105661Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"dc610c77-bc34-4e3b-9211-d16b130bb083","component":"store"} {"msg":"updated generation","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:14.194135322Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"new_generation":1,"req_id":"dc610c77-bc34-4e3b-9211-d16b130bb083","component":"store"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:14.194142872Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"PUT","req_id":"dc610c77-bc34-4e3b-9211-d16b130bb083","remote_addr":"[fd00:1122:3344:101::b]:37639","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":223,"response_code":204} {"msg":"failed to handle incoming DNS message: MessageRequest {\n header: Header {\n id: 26245,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n queries: Queries {\n queries: [\n LowerQuery {\n name: LowerName(\n Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n ),\n original: Query {\n name: Name(\"_clickhouse-admin-keeper._tcp.control-plane.oxide.internal.\"),\n query_type: SRV,\n query_class: IN,\n },\n },\n ],\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n flags: EdnsFlags {\n dnssec_ok: false,\n z: 0,\n },\n max_payload: 1232,\n options: OPT {\n options: [],\n },\n },\n ),\n} NXDOMAIN: _clickhouse-admin-keeper._tcp.control-plane.oxide.internal","v":0,"name":"dns-server","level":50,"time":"2026-05-27T01:18:15.235167529Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"peer_addr":"[fd00:1122:3344:101::b]:53009","req_id":"886ef5cf-ade5-4604-a83b-607643b398c7","component":"dns"} {"msg":"accepted connection","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:15.331060714Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","remote_addr":"[fd00:1122:3344:101::b]:54255"} {"msg":"request completed","v":0,"name":"dns-server","level":30,"time":"2026-05-27T01:18:15.331285288Z","hostname":"oxz_internal_dns_1679bc6d-cd90-41c7-864e-015cb7c7b6c3","pid":5465,"uri":"/config","method":"GET","req_id":"ba34c91f-385a-4087-9306-94b58c21aa91","remote_addr":"[fd00:1122:3344:101::b]:54255","local_addr":"[fd00:1122:3344:2::1]:5353","component":"http","latency_us":185,"response_code":200}