2024.09.28 15:28:36.985628 [ 43351 ] {} SentryWriter: Sending crash reports is disabled 2024.09.28 15:28:36.985688 [ 43351 ] {} Pipe: Pipe capacity is 1.00 MiB 2024.09.28 15:28:37.104399 [ 43351 ] {} : Starting ClickHouse Keeper 23.8.7.24(revision : 54477, git hash: 812b95e14ba8cf744bf1d70e6de607cf130a79fa, build id: 6484B75ECB16F4FF87A88E10249A45E1BDBDE2E5), PID 43351 2024.09.28 15:28:37.104437 [ 43351 ] {} Application: starting up 2024.09.28 15:28:37.104449 [ 43351 ] {} Application: OS Name = Linux, OS Version = 5.15.0-1015-aws, OS Architecture = x86_64 2024.09.28 15:28:37.107008 [ 43351 ] {} Application: Initializing DateLUT. 2024.09.28 15:28:37.107022 [ 43351 ] {} Application: Initialized DateLUT with time zone 'UTC'. 2024.09.28 15:28:37.147278 [ 43351 ] {} AsynchronousMetrics: Scanning /sys/class/thermal 2024.09.28 15:28:37.147301 [ 43351 ] {} AsynchronousMetrics: Scanning /sys/block 2024.09.28 15:28:37.147338 [ 43351 ] {} AsynchronousMetrics: Scanning /sys/devices/system/edac 2024.09.28 15:28:37.147352 [ 43351 ] {} AsynchronousMetrics: Scanning /sys/class/hwmon 2024.09.28 15:28:37.147517 [ 43351 ] {} Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously 2024.09.28 15:28:37.147531 [ 43351 ] {} KeeperDispatcher: Initializing storage dispatcher 2024.09.28 15:28:37.147740 [ 43351 ] {} KeeperContext: Keeper feature flag FILTERED_LIST: enabled 2024.09.28 15:28:37.147746 [ 43351 ] {} KeeperContext: Keeper feature flag MULTI_READ: enabled 2024.09.28 15:28:37.147749 [ 43351 ] {} KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled 2024.09.28 15:28:37.147914 [ 43351 ] {} KeeperSnapshotManager: Reading from disk LocalSnapshotDisk 2024.09.28 15:28:37.148022 [ 43351 ] {} KeeperLogStore: Reading from disk LocalLogDisk 2024.09.28 15:28:37.148045 [ 43351 ] {} KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/integration_test-191fb91badeb8f2e-test_raft_config_parsing.43348.1-oximeter-clickward-test/keeper-3/coordination/log. It's Ok if it's the first run of clickhouse-keeper. 2024.09.28 15:28:37.148156 [ 43351 ] {} KeeperLogStore: force_sync enabled 2024.09.28 15:28:37.148162 [ 43351 ] {} KeeperDispatcher: Waiting server to initialize 2024.09.28 15:28:37.148169 [ 43351 ] {} KeeperStateMachine: Totally have 0 snapshots 2024.09.28 15:28:37.148173 [ 43351 ] {} KeeperStateMachine: No existing snapshots, last committed log index 0 2024.09.28 15:28:37.148204 [ 43351 ] {} KeeperLogStore: Removing all changelogs 2024.09.28 15:28:37.148215 [ 43351 ] {} Changelog: Starting new changelog changelog_1_100000.bin.zstd 2024.09.28 15:28:37.148258 [ 43351 ] {} KeeperServer: Preprocessing 0 log entries 2024.09.28 15:28:37.148263 [ 43351 ] {} KeeperServer: Preprocessing done 2024.09.28 15:28:37.148267 [ 43351 ] {} KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk 2024.09.28 15:28:37.148658 [ 43351 ] {} RaftInstance: Raft ASIO listener initiated on :::39103, unsecured 2024.09.28 15:28:37.148686 [ 43351 ] {} KeeperStateManager: No state was read 2024.09.28 15:28:37.148699 [ 43351 ] {} RaftInstance: parameters: election timeout range 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 100000, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on 2024.09.28 15:28:37.148704 [ 43351 ] {} RaftInstance: new election timeout range: 1000 - 2000 2024.09.28 15:28:37.148720 [ 43351 ] {} RaftInstance: === INIT RAFT SERVER === commit index 0 term 0 election timer allowed log store start 1, end 0 config log idx 0, prev log idx 0 2024.09.28 15:28:37.148752 [ 43351 ] {} RaftInstance: asio client created: 0x7f5458654318 2024.09.28 15:28:37.148773 [ 43351 ] {} RaftInstance: asio client created: 0x7f5458654698 2024.09.28 15:28:37.148781 [ 43351 ] {} RaftInstance: peer 1: DC ID 0, ::1:39101, voting member, 1 peer 2: DC ID 0, ::1:39102, voting member, 1 peer 3: DC ID 0, ::1:39103, voting member, 1 my id: 3, voting_member num peers: 2 2024.09.28 15:28:37.148786 [ 43351 ] {} RaftInstance: global manager does not exist. will use local thread for commit and append 2024.09.28 15:28:37.148847 [ 43351 ] {} RaftInstance: wait for HB, for 50 + [1000, 2000] ms 2024.09.28 15:28:37.154999 [ 43406 ] {} RaftInstance: commit_cv_ sleep 2024.09.28 15:28:37.155059 [ 43407 ] {} RaftInstance: bg append_entries thread initiated 2024.09.28 15:28:37.198946 [ 43351 ] {} RaftInstance: re-schedule election timer 2024.09.28 15:28:37.198972 [ 43351 ] {} RaftInstance: server 3 started 2024.09.28 15:28:37.199013 [ 43351 ] {} RaftInstance: asio rpc session created: 0x7f545873c918 2024.09.28 15:28:37.199038 [ 43351 ] {} KeeperDispatcher: Server initialized, waiting for quorum 2024.09.28 15:28:38.307636 [ 43398 ] {} RaftInstance: receive a incoming rpc connection 2024.09.28 15:28:38.307685 [ 43398 ] {} RaftInstance: session 1 got connection from ::1:33830 (as a server) 2024.09.28 15:28:38.307764 [ 43398 ] {} RaftInstance: asio rpc session created: 0x7f5458e70018 2024.09.28 15:28:38.308042 [ 43398 ] {} RaftInstance: Receive a pre_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0 2024.09.28 15:28:38.308067 [ 43398 ] {} RaftInstance: [PRE-VOTE REQ] my role follower, from peer 2, log term: req 0 / mine 0 last idx: req 0 / mine 0, term: req 0 / mine 0 HB dead 2024.09.28 15:28:38.308078 [ 43398 ] {} RaftInstance: pre-vote decision: O (grant) 2024.09.28 15:28:38.308084 [ 43398 ] {} RaftInstance: Response back a pre_vote_response message to 2 with Accepted=1, Term=0, NextIndex=1 2024.09.28 15:28:38.311108 [ 43399 ] {} RaftInstance: Receive a request_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=1 2024.09.28 15:28:38.313755 [ 43399 ] {} RaftInstance: [BECOME FOLLOWER] term 1 2024.09.28 15:28:38.313772 [ 43399 ] {} RaftInstance: cancel existing timer 2024.09.28 15:28:38.313781 [ 43399 ] {} RaftInstance: re-schedule election timer 2024.09.28 15:28:38.313789 [ 43399 ] {} RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1 2024.09.28 15:28:38.313795 [ 43399 ] {} RaftInstance: [VOTE REQ] my role follower, from peer 2, log term: req 0 / mine 0 last idx: req 0 / mine 0, term: req 1 / mine 1 priority: target 1 / mine 1, voted_for -1 2024.09.28 15:28:38.313803 [ 43399 ] {} RaftInstance: decision: O (grant), voted_for 2, term 1 2024.09.28 15:28:38.314576 [ 43399 ] {} IDisk: Copying from LocalStateFileDisk (path: /var/tmp/omicron_tmp/integration_test-191fb91badeb8f2e-test_raft_config_parsing.43348.1-oximeter-clickward-test/keeper-3/coordination) state to LocalStateFileDisk (path: /var/tmp/omicron_tmp/integration_test-191fb91badeb8f2e-test_raft_config_parsing.43348.1-oximeter-clickward-test/keeper-3/coordination) state-OLD. 2024.09.28 15:28:38.316986 [ 43399 ] {} RaftInstance: Response back a request_vote_response message to 2 with Accepted=1, Term=1, NextIndex=1 2024.09.28 15:28:38.325008 [ 43401 ] {} RaftInstance: Receive a append_entries_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=1, CommitIndex=0 and Term=1 2024.09.28 15:28:38.325037 [ 43401 ] {} RaftInstance: from peer 2, req type: 3, req term: 1, req l idx: 0 (1), req c idx: 0, my term: 1, my role: 1 2024.09.28 15:28:38.325051 [ 43401 ] {} RaftInstance: (update) new target priority: 1 2024.09.28 15:28:38.325062 [ 43401 ] {} RaftInstance: [LOG OK] req log idx: 0, req log term: 0, my last log idx: 0, my log (0) term: 0 2024.09.28 15:28:38.325068 [ 43401 ] {} RaftInstance: [INIT] log_idx: 1, count: 0, log_store_->next_slot(): 1, req.log_entries().size(): 1 2024.09.28 15:28:38.325073 [ 43401 ] {} RaftInstance: [after SKIP] log_idx: 1, count: 0 2024.09.28 15:28:38.325077 [ 43401 ] {} RaftInstance: [after OVWR] log_idx: 1, count: 0 2024.09.28 15:28:38.325082 [ 43401 ] {} RaftInstance: append at 1, term 1, timestamp 0 2024.09.28 15:28:38.328348 [ 43393 ] {} RaftInstance: got log append completion notification: OK 2024.09.28 15:28:38.328379 [ 43401 ] {} RaftInstance: receive a config change from leader at 1 2024.09.28 15:28:38.328389 [ 43401 ] {} RaftInstance: local log idx 1, target_commit_idx 0, quick_commit_index_ 0, state_->get_commit_idx() 0 2024.09.28 15:28:38.328398 [ 43401 ] {} RaftInstance: cancel existing timer 2024.09.28 15:28:38.328407 [ 43401 ] {} RaftInstance: re-schedule election timer 2024.09.28 15:28:38.328416 [ 43401 ] {} RaftInstance: batch size hint: 0 bytes 2024.09.28 15:28:38.328420 [ 43401 ] {} RaftInstance: Response back a append_entries_response message to 2 with Accepted=1, Term=1, NextIndex=2 2024.09.28 15:28:38.328485 [ 43351 ] {} KeeperDispatcher: Quorum initialized 2024.09.28 15:28:38.328602 [ 43351 ] {} KeeperDispatcher: Dispatcher initialized 2024.09.28 15:28:38.328834 [ 43393 ] {} RaftInstance: got log append completion notification: OK 2024.09.28 15:28:38.329010 [ 43351 ] {} Context: Shutting down named sessions 2024.09.28 15:28:38.329024 [ 43351 ] {} Context: Shutting down database catalog 2024.09.28 15:28:38.329031 [ 43351 ] {} DatabaseCatalog: Shutting down system databases 2024.09.28 15:28:38.329036 [ 43351 ] {} Context: Shutting down DDLWorker 2024.09.28 15:28:38.329040 [ 43351 ] {} Context: Shutting down caches 2024.09.28 15:28:38.329087 [ 43351 ] {} KeeperDispatcher: Shutting down storage dispatcher 2024.09.28 15:28:38.329149 [ 43400 ] {} RaftInstance: Receive a append_entries_request message from 2 with LastLogIndex=1, LastLogTerm 1, EntriesLength=0, CommitIndex=1 and Term=1 2024.09.28 15:28:38.329182 [ 43400 ] {} RaftInstance: from peer 2, req type: 3, req term: 1, req l idx: 1 (0), req c idx: 1, my term: 1, my role: 1 2024.09.28 15:28:38.329196 [ 43400 ] {} RaftInstance: (update) new target priority: 1 2024.09.28 15:28:38.329203 [ 43400 ] {} RaftInstance: [LOG OK] req log idx: 1, req log term: 1, my last log idx: 1, my log (1) term: 1 2024.09.28 15:28:38.329208 [ 43400 ] {} RaftInstance: trigger commit upto 1 2024.09.28 15:28:38.329212 [ 43400 ] {} RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0 2024.09.28 15:28:38.329217 [ 43400 ] {} RaftInstance: commit_cv_ notify (local thread) 2024.09.28 15:28:38.329225 [ 43400 ] {} RaftInstance: cancel existing timer 2024.09.28 15:28:38.329242 [ 43400 ] {} RaftInstance: re-schedule election timer 2024.09.28 15:28:38.329250 [ 43400 ] {} RaftInstance: batch size hint: 0 bytes 2024.09.28 15:28:38.329255 [ 43400 ] {} RaftInstance: Response back a append_entries_response message to 2 with Accepted=1, Term=1, NextIndex=2 2024.09.28 15:28:38.329439 [ 43351 ] {} RaftInstance: shutting down raft core 2024.09.28 15:28:38.329449 [ 43351 ] {} RaftInstance: sent stop signal to the commit thread. 2024.09.28 15:28:38.329461 [ 43351 ] {} RaftInstance: asio client destroyed: 0x7f5458654698 2024.09.28 15:28:38.329748 [ 43406 ] {} RaftInstance: commit_cv_ wake up 2024.09.28 15:28:38.333039 [ 43351 ] {} RaftInstance: asio client destroyed: 0x7f5458654318 2024.09.28 15:28:38.333062 [ 43351 ] {} RaftInstance: cancelled all schedulers. 2024.09.28 15:28:38.333066 [ 43351 ] {} RaftInstance: commit thread stopped. 2024.09.28 15:28:38.333069 [ 43351 ] {} RaftInstance: all pending commit elements dropped. 2024.09.28 15:28:38.333071 [ 43351 ] {} RaftInstance: reset all pointers. 2024.09.28 15:28:38.333077 [ 43351 ] {} RaftInstance: joined terminated commit thread. 2024.09.28 15:28:38.333111 [ 43351 ] {} RaftInstance: sent stop signal to background append thread. 2024.09.28 15:28:38.333112 [ 43407 ] {} RaftInstance: bg append_entries thread terminated 2024.09.28 15:28:38.333193 [ 43351 ] {} RaftInstance: clean up auto-forwarding queue: 0 elems 2024.09.28 15:28:38.334226 [ 43351 ] {} RaftInstance: clean up auto-forwarding clients 2024.09.28 15:28:38.334235 [ 43351 ] {} RaftInstance: raft_server shutdown completed. 2024.09.28 15:28:38.334239 [ 43351 ] {} RaftInstance: manually create a snapshot on 0 2024.09.28 15:28:38.334243 [ 43351 ] {} RaftInstance: creating a snapshot for index 0 2024.09.28 15:28:38.334248 [ 43351 ] {} RaftInstance: create snapshot idx 0 log_term 0 2024.09.28 15:28:38.334253 [ 43351 ] {} KeeperStateMachine: Creating snapshot 0 2024.09.28 15:28:38.334268 [ 43351 ] {} KeeperStateMachine: Creating a snapshot during shutdown because 'create_snapshot_on_exit' is enabled. 2024.09.28 15:28:38.336728 [ 43351 ] {} KeeperStateMachine: Created persistent snapshot 0 with path snapshot_0.bin.zstd 2024.09.28 15:28:38.336737 [ 43351 ] {} KeeperStateMachine: Clearing garbage after snapshot 2024.09.28 15:28:38.336740 [ 43351 ] {} KeeperStateMachine: Cleared garbage after snapshot 2024.09.28 15:28:38.336747 [ 43351 ] {} RaftInstance: snapshot idx 0 log_term 0 created, compact the log store if needed 2024.09.28 15:28:38.336752 [ 43351 ] {} KeeperStateMachine: Uploading snapshot snapshot_0.bin.zstd during shutdown because 'upload_snapshot_on_exit' is enabled. 2024.09.28 15:28:38.336758 [ 43351 ] {} RaftInstance: create snapshot idx 0 log_term 0 done: 2506 us elapsed 2024.09.28 15:28:38.336818 [ 43400 ] {} RaftInstance: end of asio worker thread, remaining threads: 5 2024.09.28 15:28:38.336886 [ 43399 ] {} RaftInstance: end of asio worker thread, remaining threads: 3 2024.09.28 15:28:38.336871 [ 43404 ] {} RaftInstance: end of asio worker thread, remaining threads: 7 2024.09.28 15:28:38.336896 [ 43403 ] {} RaftInstance: end of asio worker thread, remaining threads: 4 2024.09.28 15:28:38.336970 [ 43405 ] {} RaftInstance: end of asio worker thread, remaining threads: 2 2024.09.28 15:28:38.336999 [ 43402 ] {} RaftInstance: end of asio worker thread, remaining threads: 1 2024.09.28 15:28:38.337042 [ 43398 ] {} RaftInstance: end of asio worker thread, remaining threads: 0 2024.09.28 15:28:38.336820 [ 43401 ] {} RaftInstance: end of asio worker thread, remaining threads: 5 2024.09.28 15:28:38.337302 [ 43351 ] {} KeeperSnapshotManagerS3: Shutting down KeeperSnapshotManagerS3 2024.09.28 15:28:38.337321 [ 43351 ] {} KeeperSnapshotManagerS3: KeeperSnapshotManagerS3 shut down 2024.09.28 15:28:38.337326 [ 43351 ] {} KeeperDispatcher: Dispatcher shut down 2024.09.28 15:28:38.337343 [ 43351 ] {} RaftInstance: asio rpc session destroyed: 0x7f545873c918 2024.09.28 15:28:38.337387 [ 43351 ] {} RaftInstance: asio rpc session destroyed: 0x7f5458e70018 2024.09.28 15:28:38.347523 [ 43351 ] {} KeeperLogStore: Changelog is shut down 2024.09.28 15:28:38.355133 [ 43351 ] {} Application: Code: 210. DB::Exception: Listen [::1]:39003 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: [::1]:39003 (version 23.8.7.24 (official build)). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6487f7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String const&, int&, String&&) @ 0x000000000c6cf2f8 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. DB::Keeper::createServer(String const&, char const*, bool, std::function&&) const @ 0x000000000c8a3176 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::Keeper::main(std::vector> const&) @ 0x000000000c8a680d in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. Poco::Util::Application::run() @ 0x0000000015b6da26 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. DB::Keeper::run() @ 0x000000000c8a375e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000015b7cb39 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. mainEntryClickHouseKeeper(int, char**) @ 0x000000000c8a2918 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. main @ 0x000000000715572f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. ? @ 0x00007f54593f5d90 in ? 10. ? @ 0x00007f54593f5e40 in ? 11. _start @ 0x000000000634fa2e in /work/oxidecomputer/omicron/out/clickhouse/clickhouse (version 23.8.7.24 (official build)) 2024.09.28 15:28:38.355194 [ 43351 ] {} Application: DB::Exception: Listen [::1]:39003 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: [::1]:39003 (version 23.8.7.24 (official build)) 2024.09.28 15:28:38.355200 [ 43351 ] {} Application: shutting down 2024.09.28 15:28:38.355202 [ 43351 ] {} Application: Uninitializing subsystem: Logging Subsystem 2024.09.28 15:28:38.355254 [ 43358 ] {} BaseDaemon: Received signal -2 2024.09.28 15:28:38.355273 [ 43358 ] {} BaseDaemon: Stop SignalListener thread