2024.09.02 19:50:10.785332 [ 207418 ] {} <Error> ZooKeeperClient: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4. (CANNOT_READ_ALL_DATA), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception<unsigned long&, String>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<String>::type>, unsigned long&, String&&) @ 0x0000000007354cb4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x0000000007354bb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)>>&&)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)>>&&)::$_1&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001384c492 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. ? @ 0x00007efcbab46ac3 in ?
6. ? @ 0x00007efcbabd8850 in ?
 (version 23.8.7.24 (official build))
2024.09.02 19:50:10.785530 [ 207290 ] {} <Warning> oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:10.785548 [ 207295 ] {} <Warning> oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:10.785598 [ 207287 ] {} <Warning> oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:10.785611 [ 207302 ] {} <Warning> oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:10.785655 [ 207297 ] {} <Warning> oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:10.787485 [ 207418 ] {} <Error> ZooKeeperClient: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b5abb2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. Poco::Net::SocketImpl::shutdown() @ 0x0000000015b5c39c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. Coordination::ZooKeeper::finalize(bool, bool, String const&) @ 0x000000001383d143 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)>>&&)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node>> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)>>&&)::$_1&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001384e087 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. ? @ 0x00007efcbab46ac3 in ?
6. ? @ 0x00007efcbabd8850 in ?
 (version 23.8.7.24 (official build))
2024.09.02 19:50:12.070704 [ 207202 ] {} <Warning> oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:12.070786 [ 207214 ] {} <Warning> oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:12.070801 [ 207220 ] {} <Warning> oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:12.070831 [ 207208 ] {} <Warning> oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:12.070874 [ 207226 ] {} <Warning> oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.09.02 19:50:12.079302 [ 207410 ] {} <Error> virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: Coordination error: Connection loss, path /clickhouse/task_queue/ddl/. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception<char const*, String const&>(int, FormatStringHelperImpl<std::type_identity<char const*>::type, std::type_identity<String const&>::type>, char const*&&, String const&) @ 0x0000000010ba7474 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. Coordination::Exception::fromPath(Coordination::Error, String const&) @ 0x0000000010ba6c48 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. zkutil::ZooKeeper::createAncestors(String const&) @ 0x00000000137f8935 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. DB::DDLWorker::initializeMainThread() @ 0x00000000116fc925 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. DB::DDLWorker::runMainThread() @ 0x00000000116e7280 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000116fd92c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. ? @ 0x00007efcbab46ac3 in ?
9. ? @ 0x00007efcbabd8850 in ?
 (version 23.8.7.24 (official build))
2024.09.02 19:50:12.192480 [ 207169 ] {0d434a24-4043-4c13-8ec7-e50b877b526f} <Error> executeQuery: Code: 244. DB::Exception: Insert failed due to zookeeper error. Please retry. Reason: Connection loss. (UNEXPECTED_ZOOKEEPER_ERROR) (version 23.8.7.24 (official build)) (from [::1]:48516) (in query: INSERT INTO oximeter.fields_i64 FORMAT JSONEachRow ), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::ZooKeeperRetriesControl::setUserError(int, String) @ 0x0000000012f32ffd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, String const&, unsigned long, bool)::'lambda'()::operator()() const @ 0x0000000012f3586a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, String const&, unsigned long, bool) @ 0x0000000012f280d8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. DB::ReplicatedMergeTreeSinkImpl<false>::finishDelayedChunk(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&) @ 0x0000000012f27270 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. DB::ReplicatedMergeTreeSinkImpl<false>::consume(DB::Chunk) @ 0x0000000012f249db in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x00000000131b33e8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::DistributedSink::writeToLocal(DB::Cluster::ShardInfo const&, DB::Block const&, unsigned long) @ 0x0000000012b19942 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::DistributedSink::writeAsyncImpl(DB::Block const&, unsigned long) @ 0x0000000012b172b7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::DistributedSink::consume(DB::Chunk) @ 0x0000000012b13c1e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
21. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
22. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x00000000131b2f62 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
23. DB::CompletedPipelineExecutor::execute() @ 0x00000000131b1812 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
24. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, std::optional<DB::FormatSettings> const&) @ 0x00000000122e9e0c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
25. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&) @ 0x00000000130fd60f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
26. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x0000000013101ee9 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
27. DB::HTTPServerConnection::run() @ 0x0000000013172ad2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
28. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5f7f4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
29. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b609f1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
30. Poco::PooledThread::run() @ 0x0000000015c97227 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
31. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c954fc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse

2024.09.02 19:50:12.194265 [ 207169 ] {0d434a24-4043-4c13-8ec7-e50b877b526f} <Error> DynamicQueryHandler: Code: 244. DB::Exception: Insert failed due to zookeeper error. Please retry. Reason: Connection loss. (UNEXPECTED_ZOOKEEPER_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::ZooKeeperRetriesControl::setUserError(int, String) @ 0x0000000012f32ffd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, String const&, unsigned long, bool)::'lambda'()::operator()() const @ 0x0000000012f3586a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. DB::ReplicatedMergeTreeSinkImpl<false>::commitPart(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, std::shared_ptr<DB::IMergeTreeDataPart>&, String const&, unsigned long, bool) @ 0x0000000012f280d8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. DB::ReplicatedMergeTreeSinkImpl<false>::finishDelayedChunk(std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&) @ 0x0000000012f27270 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. DB::ReplicatedMergeTreeSinkImpl<false>::consume(DB::Chunk) @ 0x0000000012f249db in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x00000000131b33e8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::DistributedSink::writeToLocal(DB::Cluster::ShardInfo const&, DB::Block const&, unsigned long) @ 0x0000000012b19942 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. DB::DistributedSink::writeAsyncImpl(DB::Block const&, unsigned long) @ 0x0000000012b172b7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. DB::DistributedSink::consume(DB::Chunk) @ 0x0000000012b13c1e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
17. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
18. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
19. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
20. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
21. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
22. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x00000000131b2f62 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
23. DB::CompletedPipelineExecutor::execute() @ 0x00000000131b1812 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
24. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, std::optional<DB::FormatSettings> const&) @ 0x00000000122e9e0c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
25. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&) @ 0x00000000130fd60f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
26. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x0000000013101ee9 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
27. DB::HTTPServerConnection::run() @ 0x0000000013172ad2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
28. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5f7f4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
29. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b609f1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
30. Poco::PooledThread::run() @ 0x0000000015c97227 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
31. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c954fc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
 (version 23.8.7.24 (official build))