2024.08.27 19:20:10.893422 [ 253 ] {} <Error> ZooKeeperClient: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4. (CANNOT_READ_ALL_DATA), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<unsigned long&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::type>, unsigned long&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000ac3631e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x000000000ac360fa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::receiveEvent() @ 0x0000000016bafed2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::receiveThread() @ 0x0000000016baf7d8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. std::_Function_handler<void (), ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&)::$_1&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) @ 0x0000000016bb4b98 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x0000000019092c5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32
9. _thrp_setup @ 0x0000000000114f37 in /lib/amd64/libc.so.1
 (version 23.8.7.1)
2024.08.27 19:20:10.893782 [ 105 ] {} <Warning> oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:10.893809 [ 121 ] {} <Warning> oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:10.893828 [ 136 ] {} <Warning> oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:10.894071 [ 117 ] {} <Warning> oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:10.894127 [ 108 ] {} <Warning> oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:10.894227 [ 253 ] {} <Error> ZooKeeperClient: Poco::Exception. Code: 1000, e.code() = 134, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

 (version 23.8.7.1)
2024.08.27 19:20:11.788097 [ 125 ] {} <Warning> oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:11.788140 [ 148 ] {} <Warning> oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:11.788169 [ 138 ] {} <Warning> oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:11.788260 [ 139 ] {} <Warning> oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:11.788344 [ 115 ] {} <Warning> oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:11.906723 [ 3 ] {a636843f-e7e4-4933-b559-ae08636bf57e} <Error> executeQuery: Code: 244. DB::Exception: Insert failed due to zookeeper error. Please retry. Reason: Session expired. (UNEXPECTED_ZOOKEEPER_ERROR) (version 23.8.7.1) (from [::1]:37610) (in query: INSERT INTO oximeter.fields_uuid FORMAT JSONEachRow ), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::ZooKeeperRetriesControl::setUserError(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) @ 0x0000000015a0e8de in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#1}::operator()() const @ 0x0000000015a11f03 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. void DB::ZooKeeperRetriesControl::retryLoop<DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#1}, DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#2}>(DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#1}&&, DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#2}&&) @ 0x0000000015a043b8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool) @ 0x0000000015a03df2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. DB::ReplicatedMergeTreeSinkImpl<false>::finishDelayedChunk(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&) @ 0x00000000159f86fd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. DB::ReplicatedMergeTreeSinkImpl<false>::consume(DB::Chunk) @ 0x0000000015a02e3f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x0000000015cadc28 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. std::_Function_handler<void (), DB::ExceptionKeepingTransform::work()::$_1>::_M_invoke(std::_Any_data const&) @ 0x0000000015c400d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000015c3fec5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::ExceptionKeepingTransform::work() @ 0x0000000015c3f807 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::ExecutionThreadContext::executeTask() @ 0x000000001695a188 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000015b7556c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x0000000015b75173 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::DistributedSink::writeToLocal(DB::Cluster::ShardInfo const&, DB::Block const&, unsigned long) @ 0x000000001572dc10 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. DB::DistributedSink::writeAsyncImpl(DB::Block const&, unsigned long) @ 0x000000001572a4e6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. DB::DistributedSink::consume(DB::Chunk) @ 0x00000000157295a8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x0000000015cadc28 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. std::_Function_handler<void (), DB::ExceptionKeepingTransform::work()::$_1>::_M_invoke(std::_Any_data const&) @ 0x0000000015c400d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000015c3fec5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
21. DB::ExceptionKeepingTransform::work() @ 0x0000000015c3f807 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
22. DB::ExecutionThreadContext::executeTask() @ 0x000000001695a188 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
23. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000015b7556c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
24. DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x0000000015b74f97 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
25. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000015b74d81 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
26. DB::CompletedPipelineExecutor::execute() @ 0x0000000015b73c39 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
27. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, std::optional<DB::FormatSettings> const&) @ 0x000000001518d6cc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
28. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&) @ 0x00000000168a421a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
29. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x00000000168a714c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
30. DB::HTTPServerConnection::run() @ 0x00000000168c50ab in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
31. Poco::Net::TCPServerConnection::start() @ 0x000000001954534c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
32. Poco::Net::TCPServerDispatcher::run() @ 0x0000000019545821 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
33. Poco::PooledThread::run() @ 0x00000000195aa732 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
34. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000195a81e5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
35. _thrp_setup @ 0x0000000000114f37 in /lib/amd64/libc.so.1

2024.08.27 19:20:11.907303 [ 3 ] {a636843f-e7e4-4933-b559-ae08636bf57e} <Error> DynamicQueryHandler: Code: 244. DB::Exception: Insert failed due to zookeeper error. Please retry. Reason: Session expired. (UNEXPECTED_ZOOKEEPER_ERROR), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::ZooKeeperRetriesControl::setUserError(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) @ 0x0000000015a0e8de in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#1}::operator()() const @ 0x0000000015a11f03 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. void DB::ZooKeeperRetriesControl::retryLoop<DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#1}, DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#2}>(DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#1}&&, DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool)::{lambda()#2}&&) @ 0x0000000015a043b8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool) @ 0x0000000015a03df2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. DB::ReplicatedMergeTreeSinkImpl<false>::finishDelayedChunk(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&) @ 0x00000000159f86fd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. DB::ReplicatedMergeTreeSinkImpl<false>::consume(DB::Chunk) @ 0x0000000015a02e3f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x0000000015cadc28 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. std::_Function_handler<void (), DB::ExceptionKeepingTransform::work()::$_1>::_M_invoke(std::_Any_data const&) @ 0x0000000015c400d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000015c3fec5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::ExceptionKeepingTransform::work() @ 0x0000000015c3f807 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::ExecutionThreadContext::executeTask() @ 0x000000001695a188 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000015b7556c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x0000000015b75173 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::DistributedSink::writeToLocal(DB::Cluster::ShardInfo const&, DB::Block const&, unsigned long) @ 0x000000001572dc10 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. DB::DistributedSink::writeAsyncImpl(DB::Block const&, unsigned long) @ 0x000000001572a4e6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. DB::DistributedSink::consume(DB::Chunk) @ 0x00000000157295a8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x0000000015cadc28 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. std::_Function_handler<void (), DB::ExceptionKeepingTransform::work()::$_1>::_M_invoke(std::_Any_data const&) @ 0x0000000015c400d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000015c3fec5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
21. DB::ExceptionKeepingTransform::work() @ 0x0000000015c3f807 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
22. DB::ExecutionThreadContext::executeTask() @ 0x000000001695a188 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
23. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000015b7556c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
24. DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x0000000015b74f97 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
25. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000015b74d81 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
26. DB::CompletedPipelineExecutor::execute() @ 0x0000000015b73c39 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
27. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, std::optional<DB::FormatSettings> const&) @ 0x000000001518d6cc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
28. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&) @ 0x00000000168a421a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
29. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x00000000168a714c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
30. DB::HTTPServerConnection::run() @ 0x00000000168c50ab in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
31. Poco::Net::TCPServerConnection::start() @ 0x000000001954534c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
32. Poco::Net::TCPServerDispatcher::run() @ 0x0000000019545821 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
33. Poco::PooledThread::run() @ 0x00000000195aa732 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
34. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000195a81e5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
35. _thrp_setup @ 0x0000000000114f37 in /lib/amd64/libc.so.1
 (version 23.8.7.1)