2024.08.27 19:19:56.625333 [ 244 ] {} <Error> virtual bool DB::DDLWorker::initializeMainThread(): Code: 999. Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [::1]:19001, [::1]:19002, [::1]:19003
Poco::Exception. Code: 1000, e.code() = 146, Connection refused (version 23.8.7.1), [::1]:19001
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19002 (version 23.8.7.1), [::1]:19002
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19003 (version 23.8.7.1), [::1]:19003
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19001 (version 23.8.7.1), [::1]:19001
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19002 (version 23.8.7.1), [::1]:19002
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19003 (version 23.8.7.1), [::1]:19003
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19001 (version 23.8.7.1), [::1]:19001
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19002 (version 23.8.7.1), [::1]:19002
Poco::Exception. Code: 1000, e.code() = 146, Connection refused: [::1]:19003 (version 23.8.7.1), [::1]:19003
. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x000000000aded677 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. Coordination::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(Coordination::Error, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) @ 0x0000000015562ded in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::connect(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, Poco::Timespan) @ 0x0000000016bad2ca in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::ZooKeeper(std::vector<Coordination::ZooKeeper::Node, std::allocator<Coordination::ZooKeeper::Node> > const&, zkutil::ZooKeeperArgs const&, std::shared_ptr<DB::ZooKeeperLog>, std::optional<std::function<void (unsigned long, Coordination::ZooKeeper::Node const&)> >&&) @ 0x0000000016baaecb in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. zkutil::ZooKeeper::init(zkutil::ZooKeeperArgs) @ 0x0000000016b741e9 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. zkutil::ZooKeeper::ZooKeeper(Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::ZooKeeperLog>) @ 0x0000000016b75a63 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. DB::Context::getZooKeeper() const @ 0x0000000014e80509 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. DB::DDLWorker::getAndSetZooKeeper() @ 0x0000000014ebd50a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::DDLWorker::initializeMainThread() @ 0x0000000014ecbd0c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::DDLWorker::runMainThread() @ 0x0000000014ebbd51 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::{lambda()#1}::operator()() @ 0x0000000014ecd995 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x0000000019092c5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32
15. _thrp_setup @ 0x0000000000114f37 in /lib/amd64/libc.so.1
 (version 23.8.7.1)
2024.08.27 19:20:02.270689 [ 150 ] {} <Warning> oximeter.version (a8b9cbbd-ba1f-41e3-b21a-cebda6afb17e): Log pointer of source replica 2 changed while we loading queue nodes. Will retry.
2024.08.27 19:20:12.499005 [ 132 ] {} <Warning> oximeter.version (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:12.499094 [ 168 ] {} <Warning> oximeter.fields_i64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:12.499142 [ 181 ] {} <Warning> oximeter.timeseries_schema_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:12.499053 [ 55 ] {} <Warning> oximeter.measurements_cumulativef64_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:12.499551 [ 177 ] {} <Warning> oximeter.fields_uuid_local (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2024.08.27 19:20:12.518252 [ 154 ] {} <Error> oximeter.fields_uuid_local (a9e89244-7177-439d-9410-6c84872c1664): void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<char const (&) [16]>(int, char const (&) [16]) @ 0x000000000ffe19c6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. Coordination::Exception Coordination::Exception::fromMessage<char const (&) [16]>(Coordination::Error, char const (&) [16]) @ 0x0000000016bb6e38 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x0000000016bb14ce in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (Coordination::GetResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)> >) @ 0x0000000016bb2f2d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. zkutil::ZooKeeper::asyncTryGetNoThrow(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (Coordination::WatchResponse const&)>) @ 0x0000000016b78d10 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. zkutil::ZooKeeper::getImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>) @ 0x0000000016b788e1 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. zkutil::ZooKeeper::tryGetWatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>, Coordination::Error*) @ 0x0000000016b79266 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. zkutil::ZooKeeper::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&) @ 0x0000000016b78ea0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::ReplicatedMergeTreeQueue::pullLogsToQueue(std::shared_ptr<zkutil::ZooKeeper>, std::function<void (Coordination::WatchResponse const&)>, DB::ReplicatedMergeTreeQueue::PullLogsReason) @ 0x00000000159c640b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::StorageReplicatedMergeTree::queueUpdatingTask() @ 0x000000001560a3a5 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000014a568cf in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::BackgroundSchedulePool::threadFunction() @ 0x0000000014a5967e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. std::_Function_handler<void (), ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) @ 0x0000000014a59dd8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x0000000019092c5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32
17. _thrp_setup @ 0x0000000000114f37 in /lib/amd64/libc.so.1
 (version 23.8.7.1)
2024.08.27 19:20:12.524919 [ 166 ] {} <Error> oximeter.fields_uuid_local (a9e89244-7177-439d-9410-6c84872c1664): void DB::StorageReplicatedMergeTree::mutationsUpdatingTask(): Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
2. DB::Exception::Exception<char const (&) [16]>(int, char const (&) [16]) @ 0x000000000ffe19c6 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
3. Coordination::Exception Coordination::Exception::fromMessage<char const (&) [16]>(Coordination::Error, char const (&) [16]) @ 0x0000000016bb6e38 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
4. Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x0000000016bb14ce in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
5. Coordination::ZooKeeper::list(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Coordination::ListRequestType, std::function<void (Coordination::ListResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)> >) @ 0x0000000016bb37d3 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
6. zkutil::ZooKeeper::asyncTryGetChildrenNoThrow(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)> >, Coordination::ListRequestType) @ 0x0000000016b76078 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
7. zkutil::ZooKeeper::getChildrenImpl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, Coordination::Stat*, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)> >, Coordination::ListRequestType) @ 0x0000000016b75c75 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
8. zkutil::ZooKeeper::tryGetChildrenWatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, Coordination::Stat*, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)> >, Coordination::ListRequestType) @ 0x0000000016b7679a in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
9. zkutil::ZooKeeper::getChildrenWatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Coordination::Stat*, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)> >, Coordination::ListRequestType) @ 0x0000000016b76660 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
10. DB::ReplicatedMergeTreeQueue::updateMutations(std::shared_ptr<zkutil::ZooKeeper>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)> >) @ 0x00000000159cb716 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
11. DB::StorageReplicatedMergeTree::mutationsUpdatingTask() @ 0x000000001560a701 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
12. DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x0000000014a568cf in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
13. DB::BackgroundSchedulePool::threadFunction() @ 0x0000000014a5967e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
14. std::_Function_handler<void (), ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) @ 0x0000000014a59dd8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
15. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>) @ 0x0000000019092c5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse
16. ? @ 0x000000000014d750 in /usr/gcc/13/lib/amd64/libstdc++.so.6.0.32
17. _thrp_setup @ 0x0000000000114f37 in /lib/amd64/libc.so.1
 (version 23.8.7.1)