2026.08.17 23:52:15.528014 [ 430775 ] {} ZooKeeperClient: Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4. (CANNOT_READ_ALL_DATA), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, unsigned long&, String&&) @ 0x0000000007354cb4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x0000000007354bb7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001384c492 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. ? @ 0x00007f2ed2ea5a83 in ? 6. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.528143 [ 430616 ] {} oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.528165 [ 430602 ] {} oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.528185 [ 430624 ] {} oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.528194 [ 430594 ] {} oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.528220 [ 430775 ] {} ZooKeeperClient: Poco::Exception. Code: 1000, e.code() = 107, Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below): 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x0000000015b5abb2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. Poco::Net::SocketImpl::shutdown() @ 0x0000000015b5c39c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::ZooKeeper::finalize(bool, bool, String const&) @ 0x000000001383d143 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1>(Coordination::ZooKeeper::ZooKeeper(std::vector> const&, zkutil::ZooKeeperArgs const&, std::shared_ptr, std::optional>&&)::$_1&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001384e087 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. ? @ 0x00007f2ed2ea5a83 in ? 6. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.528200 [ 430630 ] {} oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.528502 [ 430622 ] {} oximeter.fields_i64_local (4ce85b75-228a-434b-bf0a-ac32163275d8): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 242. DB::Exception: Table is in readonly mode (replica path: /clickhouse/tables/1/fields_i64_local/replicas/3). (TABLE_IS_READ_ONLY), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String const&) @ 0x00000000074d2e8d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d6ae in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. ? @ 0x00007f2ed2ea5a83 in ? 7. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.528702 [ 430601 ] {} oximeter.fields_i64_local (4ce85b75-228a-434b-bf0a-ac32163275d8): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007f2ed2ea5a83 in ? 15. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.697491 [ 430514 ] {} oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.697506 [ 430495 ] {} oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.697520 [ 430535 ] {} oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.697522 [ 430505 ] {} oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.697555 [ 430510 ] {} oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:15.697655 [ 430540 ] {} oximeter.measurements_cumulativef64_local (b3621814-53ee-4a40-975e-9577a7c29267): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 242. DB::Exception: Table is in readonly mode (replica path: /clickhouse/tables/1/measurements_cumulativef64_local/replicas/3). (TABLE_IS_READ_ONLY), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String const&) @ 0x00000000074d2e8d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d6ae in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. ? @ 0x00007f2ed2ea5a83 in ? 7. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.699931 [ 430511 ] {} oximeter.fields_uuid_local (a6697841-90b5-4b0b-8f35-655266f19b34): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Coordination error: Connection loss, path /clickhouse/tables/1/fields_uuid_local/log. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, char const*&&, String const&) @ 0x0000000010ba7474 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::fromPath(Coordination::Error, String const&) @ 0x0000000010ba6c48 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function, Coordination::Error*) @ 0x00000000137fa82f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr const&, Coordination::Error*) @ 0x00000000137fa6c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::get(String const&, Coordination::Stat*, std::shared_ptr const&) @ 0x00000000137fa518 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr, std::function, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x0000000012eddf89 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d59f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. ? @ 0x00007f2ed2ea5a83 in ? 12. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.699963 [ 430533 ] {} oximeter.fields_uuid_local (a6697841-90b5-4b0b-8f35-655266f19b34): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007f2ed2ea5a83 in ? 15. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.699982 [ 430513 ] {} oximeter.fields_i64_local (4ce85b75-228a-434b-bf0a-ac32163275d8): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007f2ed2ea5a83 in ? 15. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.700001 [ 430536 ] {} oximeter.measurements_cumulativef64_local (b3621814-53ee-4a40-975e-9577a7c29267): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007f2ed2ea5a83 in ? 15. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.700011 [ 430516 ] {} oximeter.version (c0ee7deb-8ff1-49dc-a4f8-eaaa2e6fe247): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007f2ed2ea5a83 in ? 15. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.701034 [ 430532 ] {} oximeter.version (c0ee7deb-8ff1-49dc-a4f8-eaaa2e6fe247): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::get(String const&, std::function, std::shared_ptr>) @ 0x0000000013847605 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::getImpl(String const&, String&, Coordination::Stat*, std::function) @ 0x00000000137fa0bb in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function, Coordination::Error*) @ 0x00000000137fa7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr const&, Coordination::Error*) @ 0x00000000137fa6c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::get(String const&, Coordination::Stat*, std::shared_ptr const&) @ 0x00000000137fa518 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr, std::function, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x0000000012eddcf5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d59f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007f2ed2ea5a83 in ? 15. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:15.806805 [ 430411 ] {ec6cc020-dd99-4801-ae31-b848c30897d6} executeQuery: Code: 244. DB::Exception: Insert failed due to zookeeper error. Please retry. Reason: Session expired. (UNEXPECTED_ZOOKEEPER_ERROR) (version 23.8.7.24 (official build)) (from [::1]:52116) (in query: INSERT INTO oximeter.measurements_cumulativef64 FORMAT Native), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::ZooKeeperRetriesControl::setUserError(int, String) @ 0x0000000012f32ffd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long, bool)::'lambda'()::operator()() const @ 0x0000000012f3586a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long, bool) @ 0x0000000012f280d8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. DB::ReplicatedMergeTreeSinkImpl::finishDelayedChunk(std::shared_ptr const&) @ 0x0000000012f27270 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. DB::ReplicatedMergeTreeSinkImpl::consume(DB::Chunk) @ 0x0000000012f249db in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::runStep(std::function, DB::ThreadStatus*, std::atomic*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. DB::PipelineExecutor::executeStep(std::atomic*) @ 0x00000000131b33e8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. DB::DistributedSink::writeToLocal(DB::Cluster::ShardInfo const&, DB::Block const&, unsigned long) @ 0x0000000012b19942 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. DB::DistributedSink::writeAsyncImpl(DB::Block const&, unsigned long) @ 0x0000000012b172b7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 15. DB::DistributedSink::consume(DB::Chunk) @ 0x0000000012b13c1e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 16. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 17. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 18. DB::runStep(std::function, DB::ThreadStatus*, std::atomic*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 19. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 20. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 21. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 22. DB::PipelineExecutor::executeStep(std::atomic*) @ 0x00000000131b33e8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 23. DB::TCPHandler::runImpl() @ 0x0000000013157ce6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 24. DB::TCPHandler::run() @ 0x00000000131691b9 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 25. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5f7f4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 26. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b609f1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 27. Poco::PooledThread::run() @ 0x0000000015c97227 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 28. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c954fc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 29. ? @ 0x00007f2ed2ea5a83 in ? 30. ? @ 0x00007f2ed2f37890 in ? 2026.08.17 23:52:15.806874 [ 430411 ] {ec6cc020-dd99-4801-ae31-b848c30897d6} TCPHandler: Code: 244. DB::Exception: Insert failed due to zookeeper error. Please retry. Reason: Session expired. (UNEXPECTED_ZOOKEEPER_ERROR), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::ZooKeeperRetriesControl::setUserError(int, String) @ 0x0000000012f32ffd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long, bool)::'lambda'()::operator()() const @ 0x0000000012f3586a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long, bool) @ 0x0000000012f280d8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. DB::ReplicatedMergeTreeSinkImpl::finishDelayedChunk(std::shared_ptr const&) @ 0x0000000012f27270 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. DB::ReplicatedMergeTreeSinkImpl::consume(DB::Chunk) @ 0x0000000012f249db in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::runStep(std::function, DB::ThreadStatus*, std::atomic*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. DB::PipelineExecutor::executeStep(std::atomic*) @ 0x00000000131b33e8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. DB::DistributedSink::writeToLocal(DB::Cluster::ShardInfo const&, DB::Block const&, unsigned long) @ 0x0000000012b19942 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. DB::DistributedSink::writeAsyncImpl(DB::Block const&, unsigned long) @ 0x0000000012b172b7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 15. DB::DistributedSink::consume(DB::Chunk) @ 0x0000000012b13c1e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 16. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x00000000134dc0a6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 17. void std::__function::__policy_invoker::__call_impl>(std::__function::__policy_storage const*) @ 0x0000000013417e8b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 18. DB::runStep(std::function, DB::ThreadStatus*, std::atomic*) @ 0x0000000013417bbc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 19. DB::ExceptionKeepingTransform::work() @ 0x00000000134172b3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 20. DB::ExecutionThreadContext::executeTask() @ 0x00000000131bcdfa in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 21. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x00000000131b3c30 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 22. DB::PipelineExecutor::executeStep(std::atomic*) @ 0x00000000131b33e8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 23. DB::TCPHandler::runImpl() @ 0x0000000013157ce6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 24. DB::TCPHandler::run() @ 0x00000000131691b9 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 25. Poco::Net::TCPServerConnection::start() @ 0x0000000015b5f7f4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 26. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b609f1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 27. Poco::PooledThread::run() @ 0x0000000015c97227 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 28. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c954fc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 29. ? @ 0x00007f2ed2ea5a83 in ? 30. ? @ 0x00007f2ed2f37890 in ? 2026.08.17 23:52:26.923354 [ 430646 ] {} oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:26.923363 [ 430567 ] {} oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:26.923375 [ 430560 ] {} oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:26.923380 [ 430597 ] {} oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:26.923386 [ 430608 ] {} oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session. 2026.08.17 23:52:26.923492 [ 430637 ] {} oximeter.fields_uuid_local (a6697841-90b5-4b0b-8f35-655266f19b34): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 242. DB::Exception: Table is in readonly mode (replica path: /clickhouse/tables/1/fields_uuid_local/replicas/3). (TABLE_IS_READ_ONLY), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String const&) @ 0x00000000074d2e8d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d6ae in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. ? @ 0x00007f2ed2ea5a83 in ? 7. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:26.924682 [ 430620 ] {} oximeter.measurements_cumulativef64_local (b3621814-53ee-4a40-975e-9577a7c29267): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Coordination error: Operation timeout. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type>, char const*&&) @ 0x000000000af9d01c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception::Exception(Coordination::Error) @ 0x00000000137efee7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. zkutil::KeeperMultiException::check(Coordination::Error, std::vector, std::allocator>> const&, std::vector, std::allocator>> const&) @ 0x00000000137fbc3d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. zkutil::MultiReadResponses zkutil::ZooKeeper::multiRead>(std::__wrap_iter, std::__wrap_iter, std::function (String const&)>, std::function (String const&)>) @ 0x000000000c8cb70b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::MultiReadResponses zkutil::ZooKeeper::get>(std::__wrap_iter, std::__wrap_iter) @ 0x000000000c8cb486 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr, std::function, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x0000000012edf492 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d59f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. ? @ 0x00007f2ed2ea5a83 in ? 12. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:26.924718 [ 430580 ] {} oximeter.fields_i64_local (4ce85b75-228a-434b-bf0a-ac32163275d8): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::list(String const&, Coordination::ListRequestType, std::function, std::shared_ptr>) @ 0x00000000138481b1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(String const&, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f728e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::getChildrenImpl(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f6d96 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGetChildrenWatch(String const&, std::vector>&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7a7b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::getChildrenWatch(String const&, Coordination::Stat*, std::shared_ptr>, Coordination::ListRequestType) @ 0x00000000137f7971 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr, std::shared_ptr>) @ 0x0000000012ee47c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_2, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289dec0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl, StrongTypedef, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef, StrongTypedef, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000110c95d1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 13. void* std::__thread_proxy[abi:v15000]>, void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000c7298c4 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 14. ? @ 0x00007f2ed2ea5a83 in ? 15. ? @ 0x00007f2ed2f37890 in ? (version 23.8.7.24 (official build)) 2026.08.17 23:52:26.924721 [ 430581 ] {} oximeter.fields_i64_local (4ce85b75-228a-434b-bf0a-ac32163275d8): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, char const (&) [16]) @ 0x000000000c63f980 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. Coordination::Exception Coordination::Exception::fromMessage(Coordination::Error, char const (&) [16]) @ 0x0000000013817dd3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x00000000138452ec in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Coordination::ZooKeeper::get(String const&, std::function, std::shared_ptr>) @ 0x0000000013847605 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. zkutil::ZooKeeper::getImpl(String const&, String&, Coordination::Stat*, std::function) @ 0x00000000137fa0bb in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function, Coordination::Error*) @ 0x00000000137fa7df in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr const&, Coordination::Error*) @ 0x00000000137fa6c8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. zkutil::ZooKeeper::get(String const&, Coordination::Stat*, std::shared_ptr const&) @ 0x00000000137fa518 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr, std::function, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x0000000012eddcf5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. void std::__function::__policy_invoker::__call_impl, String const&, DB::MergeTreeData::MergingParams const&, std::unique_ptr>, bool, DB::RenamingRestrictions, bool)::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x000000001289d59f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 11. DB::BackgroundSchedulePool::threadFunction() @ 0x00000000110c859f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 12. void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl