2026.08.17 23:52:06.049290 [ 425097 ] {} virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19003, [::1]:19002, [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr) @ 0x00000000137f66f9 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::Context::getZooKeeper() const @ 0x000000001166afca in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::DDLWorker::getAndSetZooKeeper() @ 0x00000000116ec1ed in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::DDLWorker::initializeMainThread() @ 0x00000000116fc81b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::DDLWorker::runMainThread() @ 0x00000000116e7280 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000116fd92c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. ? @ 0x00007fd83ba48a83 in ? 13. ? @ 0x00007fd83bada890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:13.034091 [ 424739 ] {} oximeter.timeseries_schema_local (05a0062d-b532-435a-91d7-b9da99374f1d): Log pointer of source replica 2 changed while we loading queue nodes. Will retry. 2026.08.17 23:52:15.527992 [ 432271 ] {} ZooKeeperClient: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4. (CANNOT_READ_ALL_DATA), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, unsigned long&, String&&) @ 0x0000000007354cb4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x0000000007354bb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001384c492 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. ? @ 0x00007feb32c9aa83 in ? 6. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.528987 [ 432105 ] {} oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.529034 [ 432087 ] {} oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.529010 [ 432029 ] {} oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.529081 [ 432095 ] {} oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.529072 [ 432093 ] {} oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.529162 [ 432271 ] {} ZooKeeperClient: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below): 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b5abb2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. Poco::Net::SocketImpl::shutdown() @ 0x0000000015b5c39c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::ZooKeeper::finalize(bool, bool, String const&) @ 0x000000001383d143 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001384e087 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. ? @ 0x00007feb32c9aa83 in ? 6. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:16.925401 [ 432206 ] {} oximeter.fields_uuid_local (eaa5beef-895d-41af-915b-bd6fb56613c4): auto DB::StorageReplicatedMergeTree::processQueueEntry(ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(LogEntryPtr &) const: Code: 999. Coordination::Exception: Coordination error: Operation timeout. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, char const*&&) @ 0x000000000af9d01c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error) @ 0x00000000137efee7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. zkutil::ZooKeeper::tryMulti(std::vector, std::allocator>> const&, std::vector, std::allocator>>&) @ 0x00000000137fbcc2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. DB::StorageReplicatedMergeTree::checkPartChecksumsAndCommit(DB::MergeTreeData::Transaction&, std::shared_ptr const&, std::optional, bool) @ 0x000000001279544d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. DB::StorageReplicatedMergeTree::fetchPart(String const&, std::shared_ptr const&, String const&, String const&, bool, unsigned long, std::shared_ptr, bool) @ 0x00000000127be9cc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&, bool) @ 0x00000000127b3310 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&) @ 0x0000000012799da6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. bool std::__function::__policy_invoker&)>::__call_impl)::$_1, bool (std::shared_ptr&)>>(std::__function::__policy_storage const*, std::shared_ptr&) @ 0x000000001289f69f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::processEntry(std::function ()>, std::shared_ptr&, std::function&)>) @ 0x0000000012efc206 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. DB::StorageReplicatedMergeTree::processQueueEntry(std::shared_ptr) @ 0x00000000127e0c3c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::ExecutableLambdaAdapter::executeStep() @ 0x00000000128a01f2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. DB::MergeTreeBackgroundExecutor::threadFunction() @ 0x0000000012be88e0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. ThreadPoolImpl>::worker(std::__list_iterator, void*>) @ 0x000000000c7275be in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c72b07c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 15. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 16. ? @ 0x00007feb32c9aa83 in ? 17. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:17.444076 [ 432076 ] {} oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:17.444084 [ 432039 ] {} oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:17.444096 [ 432043 ] {} oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:17.444132 [ 432049 ] {} oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:17.444162 [ 432031 ] {} oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:17.444492 [ 432103 ] {} oximeter.fields_i64_local (2d79911c-8882-4309-a914-7e64c31972ae): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::get(String const&, std::function, std::shared_ptr>) @ 0x0000000013847605 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::getImpl(String const&, String&, Coordination::Stat*, std::function) @ 0x00000000137fa0bb in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function, Coordination::Error*) @ 0x00000000137fa7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr const&, Coordination::Error*) @ 0x00000000137fa6c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::get(String const&, Coordination::Stat*, std::shared_ptr const&) @ 0x00000000137fa518 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr, std::function, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x0000000012eddcf5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d59f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007feb32c9aa83 in ? 15. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:17.445392 [ 432050 ] {} oximeter.measurements_cumulativef64_local (24dea9d7-064d-49c2-9599-9587402bf78f): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Coordination error: Connection loss. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, char const*&&) @ 0x000000000af9d01c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error) @ 0x00000000137efee7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. zkutil::KeeperMultiException::check(Coordination::Error, std::vector, std::allocator>> const&, std::vector, std::allocator>> const&) @ 0x00000000137fbc3d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr, std::function, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x0000000012ee236f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d59f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. ? @ 0x00007feb32c9aa83 in ? 10. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:17.445495 [ 432044 ] {} oximeter.measurements_cumulativef64_local (24dea9d7-064d-49c2-9599-9587402bf78f): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007feb32c9aa83 in ? 15. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:17.445514 [ 432075 ] {} oximeter.fields_uuid_local (eaa5beef-895d-41af-915b-bd6fb56613c4): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007feb32c9aa83 in ? 15. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:17.445538 [ 432023 ] {} oximeter.fields_uuid_local (eaa5beef-895d-41af-915b-bd6fb56613c4): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::get(String const&, std::function, std::shared_ptr>) @ 0x0000000013847605 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::getImpl(String const&, String&, Coordination::Stat*, std::function) @ 0x00000000137fa0bb in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function, Coordination::Error*) @ 0x00000000137fa7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr const&, Coordination::Error*) @ 0x00000000137fa6c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::get(String const&, Coordination::Stat*, std::shared_ptr const&) @ 0x00000000137fa518 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr, std::function, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x0000000012eddcf5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d59f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007feb32c9aa83 in ? 15. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:17.445554 [ 432050 ] {} oximeter.measurements_cumulativef64_local (24dea9d7-064d-49c2-9599-9587402bf78f): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 242. DB::Exception: Table is in readonly mode (replica path: /clickhouse/tables/1/measurements_cumulativef64_local/replicas/1). (TABLE_IS_READ_ONLY), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String const&) @ 0x00000000074d2e8d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d6ae in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. ? @ 0x00007feb32c9aa83 in ? 7. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:17.445547 [ 432107 ] {} oximeter.fields_i64_local (2d79911c-8882-4309-a914-7e64c31972ae): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007feb32c9aa83 in ? 15. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:27.452747 [ 432274 ] {} ConfigReloader: Error updating configuration from '/var/tmp/omicron_tmp/integration_test-642120957f655e64-test_cluster.423860.1-oximeter-clickward-test/clickhouse-1/clickhouse-config.xml' config.: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19003, [::1]:19001 Code: 209. DB::NetException: Timeout exceeded while reading from socket ([::1]:19003, 10000 ms): while receiving handshake from ZooKeeper. (SOCKET_TIMEOUT) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, String const&, std::shared_ptr) @ 0x00000000137f66f9 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::reloadZooKeeperIfChangedImpl(Poco::AutoPtr const&, String const&, std::shared_ptr&, std::shared_ptr, bool) @ 0x000000001167114c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::Context::reloadZooKeeperIfChanged(Poco::AutoPtr const&) const @ 0x0000000011670f31 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::Server::main(std::vector> const&)::$_4::operator()(Poco::AutoPtr, bool) const @ 0x000000000c6ffe87 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. void std::__function::__policy_invoker, bool)>::__call_impl> const&)::$_4, void (Poco::AutoPtr, bool)>>(std::__function::__policy_storage const*, Poco::AutoPtr&&, bool) @ 0x000000000c6fda06 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) @ 0x00000000137cceec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::ConfigReloader::run() @ 0x00000000137ce1a0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(void (DB::ConfigReloader::*&&)(), DB::ConfigReloader*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000137ce9cc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007feb32c9aa83 in ? 15. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:27.452817 [ 432039 ] {} oximeter.version (ReplicatedMergeTreeRestartingThread): Failed to establish a new ZK connection. Will try again: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19003, [::1]:19001, [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(zkutil::ZooKeeperArgs const&, std::shared_ptr) @ 0x00000000137f5ff0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::Context::getZooKeeper() const @ 0x000000001166b47c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::StorageReplicatedMergeTree::setZooKeeper() @ 0x000000001275b85e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::ReplicatedMergeTreeRestartingThread::run() @ 0x0000000012f11640 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. ? @ 0x00007feb32c9aa83 in ? 13. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:27.452826 [ 432043 ] {} oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): Failed to establish a new ZK connection. Will try again: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19001, [::1]:19003, [::1]:19002 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(zkutil::ZooKeeperArgs const&, std::shared_ptr) @ 0x00000000137f5ff0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::Context::getZooKeeper() const @ 0x000000001166b47c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::StorageReplicatedMergeTree::setZooKeeper() @ 0x000000001275b85e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::ReplicatedMergeTreeRestartingThread::run() @ 0x0000000012f11640 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. ? @ 0x00007feb32c9aa83 in ? 13. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:27.453055 [ 432031 ] {} oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): Failed to establish a new ZK connection. Will try again: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19003, [::1]:19001, [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(zkutil::ZooKeeperArgs const&, std::shared_ptr) @ 0x00000000137f5ff0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::Context::getZooKeeper() const @ 0x000000001166b47c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::StorageReplicatedMergeTree::setZooKeeper() @ 0x000000001275b85e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::ReplicatedMergeTreeRestartingThread::run() @ 0x0000000012f11640 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. ? @ 0x00007feb32c9aa83 in ? 13. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:27.453075 [ 432031 ] {} oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): Table was in readonly mode. Will try to activate it. 2026.08.17 23:52:27.453487 [ 432076 ] {} oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): Failed to establish a new ZK connection. Will try again: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19003, [::1]:19001, [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(zkutil::ZooKeeperArgs const&, std::shared_ptr) @ 0x00000000137f5ff0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::Context::getZooKeeper() const @ 0x000000001166b47c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::StorageReplicatedMergeTree::setZooKeeper() @ 0x000000001275b85e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::ReplicatedMergeTreeRestartingThread::run() @ 0x0000000012f11640 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. ? @ 0x00007feb32c9aa83 in ? 13. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:27.453948 [ 432049 ] {} oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): Failed to establish a new ZK connection. Will try again: Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19002, [::1]:19003, [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(zkutil::ZooKeeperArgs const&, std::shared_ptr) @ 0x00000000137f5ff0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::Context::getZooKeeper() const @ 0x000000001166b47c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. DB::StorageReplicatedMergeTree::setZooKeeper() @ 0x000000001275b85e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::ReplicatedMergeTreeRestartingThread::run() @ 0x0000000012f11640 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. ? @ 0x00007feb32c9aa83 in ? 13. ? @ 0x00007feb32d2c890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:27.453971 [ 432049 ] {} oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): Table was in readonly mode. Will try to activate it. 2026.08.17 23:52:27.454440 [ 432273 ] {} virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19003, [::1]:19001, [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 23.8.7.24 (official build)), [::1]:19003 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19001 Poco::Exception. Code: 1000, e.code() = 111, Connection refused (version 23.8.7.24 (official build)), [::1]:19002 . (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&) @ 0x00000000074d294d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error, FormatStringHelperImpl::type>, String&) @ 0x00000000126eb50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&) @ 0x0000000013841f00 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x00000000137f31d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::ZooKeeper(zkutil::ZooKeeperArgs const&, std::shared_ptr