| 1 | 2026.08.17 23:52:54.328491 [ 434106 ] {} <Information> SentryWriter: Sending crash reports is disabled | ||
| 2 | 2026.08.17 23:52:54.328515 [ 434106 ] {} <Trace> Pipe: Pipe capacity is 1.00 MiB | ||
| 3 | 2026.08.17 23:52:54.359403 [ 434106 ] {} <Information> : Starting ClickHouse Keeper 23.8.7.24(revision : 54477, git hash: 812b95e14ba8cf744bf1d70e6de607cf130a79fa, build id: 6484B75ECB16F4FF87A88E10249A45E1BDBDE2E5), PID 434106 | ||
| 4 | 2026.08.17 23:52:54.359426 [ 434106 ] {} <Information> Application: starting up | ||
| 5 | 2026.08.17 23:52:54.359433 [ 434106 ] {} <Information> Application: OS Name = Linux, OS Version = 5.15.0-1015-aws, OS Architecture = x86_64 | ||
| 6 | 2026.08.17 23:52:54.362624 [ 434106 ] {} <Debug> Application: Initializing DateLUT. | ||
| 7 | 2026.08.17 23:52:54.362629 [ 434106 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'. | ||
| 8 | 2026.08.17 23:52:54.383162 [ 434106 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal | ||
| 9 | 2026.08.17 23:52:54.383175 [ 434106 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block | ||
| 10 | 2026.08.17 23:52:54.383206 [ 434106 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac | ||
| 11 | 2026.08.17 23:52:54.383219 [ 434106 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon | ||
| 12 | 2026.08.17 23:52:54.383333 [ 434106 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously | ||
| 13 | 2026.08.17 23:52:54.383339 [ 434106 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher | ||
| 14 | 2026.08.17 23:52:54.383522 [ 434106 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled | ||
| 15 | 2026.08.17 23:52:54.383525 [ 434106 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled | ||
| 16 | 2026.08.17 23:52:54.383527 [ 434106 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled | ||
| 17 | 2026.08.17 23:52:54.383605 [ 434106 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk | ||
| 18 | 2026.08.17 23:52:54.383653 [ 434106 ] {} <Trace> KeeperLogStore: Reading from disk LocalLogDisk | ||
| 19 | 2026.08.17 23:52:54.383661 [ 434106 ] {} <Warning> KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/integration_test-642120957f655e64-test_cluster.434101.1-oximeter-clickward-test/keeper-1/coordination/log. It's Ok if it's the first run of clickhouse-keeper. | ||
| 20 | 2026.08.17 23:52:54.383783 [ 434106 ] {} <Information> KeeperLogStore: force_sync enabled | ||
| 21 | 2026.08.17 23:52:54.383801 [ 434106 ] {} <Debug> KeeperDispatcher: Waiting server to initialize | ||
| 22 | 2026.08.17 23:52:54.383806 [ 434106 ] {} <Debug> KeeperStateMachine: Totally have 0 snapshots | ||
| 23 | 2026.08.17 23:52:54.383808 [ 434106 ] {} <Debug> KeeperStateMachine: No existing snapshots, last committed log index 0 | ||
| 24 | 2026.08.17 23:52:54.383836 [ 434106 ] {} <Warning> KeeperLogStore: Removing all changelogs | ||
| 25 | 2026.08.17 23:52:54.383844 [ 434106 ] {} <Trace> Changelog: Starting new changelog changelog_1_100000.bin.zstd | ||
| 26 | 2026.08.17 23:52:54.383864 [ 434106 ] {} <Information> KeeperServer: Preprocessing 0 log entries | ||
| 27 | 2026.08.17 23:52:54.383866 [ 434106 ] {} <Information> KeeperServer: Preprocessing done | ||
| 28 | 2026.08.17 23:52:54.383869 [ 434106 ] {} <Information> KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk | ||
| 29 | 2026.08.17 23:52:54.384321 [ 434106 ] {} <Error> RaftInstance: got exception: bind: Address already in use [system:98] on port 19101 | ||
| 30 | 2026.08.17 23:52:54.385758 [ 434106 ] {} <Error> void DB::KeeperDispatcher::initialize(const Poco::Util::AbstractConfiguration &, bool, bool, const MultiVersion<Macros>::Version &): Code: 568. DB::Exception: Cannot create interserver listener on port 19101. (RAFT_ERROR), Stack trace (when copying this message, always include the lines below): | ||
| 31 | |||
| 32 | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 33 | 1. DB::Exception::Exception<int>(int, FormatStringHelperImpl<std::type_identity<int>::type>, int&&) @ 0x000000000cf22a5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 34 | 2. DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) @ 0x00000000136faf3d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 35 | 3. DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) @ 0x00000000136fe39e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 36 | 4. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x00000000136df8d8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 37 | 5. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000001166ff82 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 38 | 6. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000c8a65da in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 39 | 7. Poco::Util::Application::run() @ 0x0000000015b6da26 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 40 | 8. DB::Keeper::run() @ 0x000000000c8a375e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 41 | 9. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000015b7cb39 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 42 | 10. mainEntryClickHouseKeeper(int, char**) @ 0x000000000c8a2918 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 43 | 11. main @ 0x000000000715572f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 44 | 12. ? @ 0x00007f62c0e34d90 in ? | ||
| 45 | 13. ? @ 0x00007f62c0e34e40 in ? | ||
| 46 | 14. _start @ 0x000000000634fa2e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 47 | (version 23.8.7.24 (official build)) | ||
| 48 | 2026.08.17 23:52:54.385810 [ 434106 ] {} <Trace> Context: Shutting down named sessions | ||
| 49 | 2026.08.17 23:52:54.385814 [ 434106 ] {} <Trace> Context: Shutting down database catalog | ||
| 50 | 2026.08.17 23:52:54.385817 [ 434106 ] {} <Trace> DatabaseCatalog: Shutting down system databases | ||
| 51 | 2026.08.17 23:52:54.385820 [ 434106 ] {} <Trace> Context: Shutting down DDLWorker | ||
| 52 | 2026.08.17 23:52:54.385822 [ 434106 ] {} <Trace> Context: Shutting down caches | ||
| 53 | 2026.08.17 23:52:54.385861 [ 434106 ] {} <Debug> KeeperDispatcher: Shutting down storage dispatcher | ||
| 54 | 2026.08.17 23:52:54.385892 [ 434106 ] {} <Information> KeeperServer: RAFT doesn't start, shutdown not required | ||
| 55 | 2026.08.17 23:52:54.386287 [ 434162 ] {} <Warning> KeeperLogStore: Raft server is not set in LogStore. | ||
| 56 | 2026.08.17 23:52:54.386322 [ 434106 ] {} <Debug> KeeperSnapshotManagerS3: Shutting down KeeperSnapshotManagerS3 | ||
| 57 | 2026.08.17 23:52:54.386334 [ 434106 ] {} <Information> KeeperSnapshotManagerS3: KeeperSnapshotManagerS3 shut down | ||
| 58 | 2026.08.17 23:52:54.386336 [ 434106 ] {} <Debug> KeeperDispatcher: Dispatcher shut down | ||
| 59 | 2026.08.17 23:52:54.386421 [ 434174 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 12 | ||
| 60 | 2026.08.17 23:52:54.386426 [ 434172 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 14 | ||
| 61 | 2026.08.17 23:52:54.386506 [ 434192 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 8 | ||
| 62 | 2026.08.17 23:52:54.386510 [ 434182 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 9 | ||
| 63 | 2026.08.17 23:52:54.386491 [ 434186 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 10 | ||
| 64 | 2026.08.17 23:52:54.386577 [ 434200 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 2 | ||
| 65 | 2026.08.17 23:52:54.386597 [ 434170 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 15 | ||
| 66 | 2026.08.17 23:52:54.386656 [ 434204 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 4 | ||
| 67 | 2026.08.17 23:52:54.386662 [ 434189 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 5 | ||
| 68 | 2026.08.17 23:52:54.386546 [ 434180 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 11 | ||
| 69 | 2026.08.17 23:52:54.386717 [ 434198 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 7 | ||
| 70 | 2026.08.17 23:52:54.386770 [ 434195 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 0 | ||
| 71 | 2026.08.17 23:52:54.386782 [ 434178 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 1 | ||
| 72 | 2026.08.17 23:52:54.386783 [ 434185 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 6 | ||
| 73 | 2026.08.17 23:52:54.386789 [ 434202 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 3 | ||
| 74 | 2026.08.17 23:52:54.386794 [ 434176 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 13 | ||
| 75 | 2026.08.17 23:52:54.386912 [ 434106 ] {} <Warning> KeeperLogStore: Changelog is shut down | ||
| 76 | 2026.08.17 23:52:54.387071 [ 434106 ] {} <Error> Application: Code: 568. DB::Exception: Cannot create interserver listener on port 19101. (RAFT_ERROR), Stack trace (when copying this message, always include the lines below): | ||
| 77 | |||
| 78 | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 79 | 1. DB::Exception::Exception<int>(int, FormatStringHelperImpl<std::type_identity<int>::type>, int&&) @ 0x000000000cf22a5c in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 80 | 2. DB::KeeperServer::launchRaftServer(Poco::Util::AbstractConfiguration const&, bool) @ 0x00000000136faf3d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 81 | 3. DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) @ 0x00000000136fe39e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 82 | 4. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x00000000136df8d8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 83 | 5. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000001166ff82 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 84 | 6. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000c8a65da in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 85 | 7. Poco::Util::Application::run() @ 0x0000000015b6da26 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 86 | 8. DB::Keeper::run() @ 0x000000000c8a375e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 87 | 9. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000015b7cb39 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 88 | 10. mainEntryClickHouseKeeper(int, char**) @ 0x000000000c8a2918 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 89 | 11. main @ 0x000000000715572f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 90 | 12. ? @ 0x00007f62c0e34d90 in ? | ||
| 91 | 13. ? @ 0x00007f62c0e34e40 in ? | ||
| 92 | 14. _start @ 0x000000000634fa2e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 93 | (version 23.8.7.24 (official build)) | ||
| 94 | 2026.08.17 23:52:54.387096 [ 434106 ] {} <Error> Application: DB::Exception: Cannot create interserver listener on port 19101 | ||
| 95 | 2026.08.17 23:52:54.387100 [ 434106 ] {} <Information> Application: shutting down | ||
| 96 | 2026.08.17 23:52:54.387102 [ 434106 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem | ||
| 97 | 2026.08.17 23:52:54.387136 [ 434118 ] {} <Trace> BaseDaemon: Received signal -2 | ||
| 98 | 2026.08.17 23:52:54.387144 [ 434118 ] {} <Information> BaseDaemon: Stop SignalListener thread | ||