2024.09.25 20:43:43.317264 [ 1 ] {} : Starting ClickHouse Keeper 23.8.7.1(revision : 54477, git hash: b6a391579f09dde47a1aa5602e154e7fcde80d1e, build id: ), PID 10493 2024.09.25 20:43:43.318919 [ 1 ] {} Application: starting up 2024.09.25 20:43:43.319057 [ 1 ] {} Application: OS Name = SunOS, OS Version = 5.11, OS Architecture = i86pc 2024.09.25 20:43:43.341329 [ 1 ] {} Application: Initializing DateLUT. 2024.09.25 20:43:43.341377 [ 1 ] {} Application: Initialized DateLUT with time zone 'UTC'. 2024.09.25 20:43:43.397565 [ 1 ] {} Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously 2024.09.25 20:43:43.397692 [ 1 ] {} KeeperDispatcher: Initializing storage dispatcher 2024.09.25 20:43:43.398380 [ 1 ] {} KeeperContext: Keeper feature flag FILTERED_LIST: enabled 2024.09.25 20:43:43.398403 [ 1 ] {} KeeperContext: Keeper feature flag MULTI_READ: enabled 2024.09.25 20:43:43.398430 [ 1 ] {} KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled 2024.09.25 20:43:43.398769 [ 1 ] {} KeeperSnapshotManager: Reading from disk LocalSnapshotDisk 2024.09.25 20:43:43.399209 [ 1 ] {} KeeperLogStore: Reading from disk LocalLogDisk 2024.09.25 20:43:43.399296 [ 1 ] {} KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/integration_test-13fe08fd3bd99210-test_raft_config_parsing.10491.1-oximeter-clickward-test/keeper-1/coordination/log. It's Ok if it's the first run of clickhouse-keeper. 2024.09.25 20:43:43.399540 [ 1 ] {} KeeperLogStore: force_sync enabled 2024.09.25 20:43:43.399559 [ 1 ] {} KeeperDispatcher: Waiting server to initialize 2024.09.25 20:43:43.399579 [ 1 ] {} KeeperStateMachine: Totally have 0 snapshots 2024.09.25 20:43:43.399588 [ 1 ] {} KeeperStateMachine: No existing snapshots, last committed log index 0 2024.09.25 20:43:43.399690 [ 1 ] {} KeeperLogStore: Removing all changelogs 2024.09.25 20:43:43.399823 [ 1 ] {} Changelog: Starting new changelog changelog_1_100000.bin.zstd 2024.09.25 20:43:43.399947 [ 1 ] {} KeeperServer: Preprocessing 0 log entries 2024.09.25 20:43:43.399969 [ 1 ] {} KeeperServer: Preprocessing done 2024.09.25 20:43:43.399979 [ 1 ] {} KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk 2024.09.25 20:43:43.401678 [ 1 ] {} RaftInstance: Raft ASIO listener initiated on :::39101, unsecured 2024.09.25 20:43:43.401841 [ 1 ] {} KeeperStateManager: No state was read 2024.09.25 20:43:43.401895 [ 1 ] {} RaftInstance: parameters: election timeout range 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 100000, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on 2024.09.25 20:43:43.401914 [ 1 ] {} RaftInstance: new election timeout range: 1000 - 2000 2024.09.25 20:43:43.401990 [ 1 ] {} RaftInstance: === INIT RAFT SERVER === commit index 0 term 0 election timer allowed log store start 1, end 0 config log idx 0, prev log idx 0 2024.09.25 20:43:43.402237 [ 1 ] {} RaftInstance: asio client created: 1d86ac30 2024.09.25 20:43:43.402280 [ 1 ] {} RaftInstance: asio client created: 1d87d690 2024.09.25 20:43:43.402293 [ 1 ] {} RaftInstance: peer 1: DC ID 0, ::1:39101, voting member, 1 peer 2: DC ID 0, ::1:39102, voting member, 1 peer 3: DC ID 0, ::1:39103, voting member, 1 my id: 1, voting_member num peers: 2 2024.09.25 20:43:43.402376 [ 1 ] {} RaftInstance: global manager does not exist. will use local thread for commit and append 2024.09.25 20:43:43.402510 [ 1 ] {} RaftInstance: wait for HB, for 50 + [1000, 2000] ms 2024.09.25 20:43:43.403023 [ 21 ] {} RaftInstance: bg append_entries thread initiated 2024.09.25 20:43:43.454491 [ 20 ] {} RaftInstance: commit_cv_ sleep 2024.09.25 20:43:43.460954 [ 1 ] {} RaftInstance: re-schedule election timer 2024.09.25 20:43:43.460997 [ 1 ] {} RaftInstance: server 1 started 2024.09.25 20:43:43.461040 [ 1 ] {} RaftInstance: asio rpc session created: 1d86a580 2024.09.25 20:43:43.461148 [ 1 ] {} KeeperDispatcher: Server initialized, waiting for quorum 2024.09.25 20:43:44.521364 [ 13 ] {} RaftInstance: election timeout 2024.09.25 20:43:44.521426 [ 13 ] {} RaftInstance: Election timeout, initiate leader election 2024.09.25 20:43:44.521481 [ 13 ] {} RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1 2024.09.25 20:43:44.521517 [ 13 ] {} RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done 2024.09.25 20:43:44.521589 [ 13 ] {} RaftInstance: [PRE-VOTE INIT] my id 1, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1) 2024.09.25 20:43:44.521633 [ 13 ] {} RaftInstance: send req 1 -> 3, type pre_vote_request 2024.09.25 20:43:44.521662 [ 13 ] {} RaftInstance: socket 1d87d690 to ::1:39103 is not opened yet 2024.09.25 20:43:44.521800 [ 13 ] {} RaftInstance: send req 1 -> 2, type pre_vote_request 2024.09.25 20:43:44.521835 [ 13 ] {} RaftInstance: socket 1d86ac30 to ::1:39102 is not opened yet 2024.09.25 20:43:44.521849 [ 13 ] {} RaftInstance: cancel existing timer 2024.09.25 20:43:44.521859 [ 13 ] {} RaftInstance: re-schedule election timer 2024.09.25 20:43:44.522553 [ 18 ] {} RaftInstance: 1d87d690 connected to ::1:39103 (as a client) 2024.09.25 20:43:44.522689 [ 18 ] {} RaftInstance: 1d86ac30 connected to ::1:39102 (as a client) 2024.09.25 20:43:44.523073 [ 18 ] {} RaftInstance: resp of req 1 -> 3, type pre_vote_request, OK 2024.09.25 20:43:44.523100 [ 18 ] {} RaftInstance: Receive a pre_vote_response message from peer 3 with Result=1, Term=0, NextIndex=1 2024.09.25 20:43:44.523102 [ 12 ] {} RaftInstance: resp of req 1 -> 2, type pre_vote_request, OK 2024.09.25 20:43:44.523110 [ 18 ] {} RaftInstance: src: 3, dst: 1, resp->get_term(): 0 2024.09.25 20:43:44.523133 [ 18 ] {} RaftInstance: [PRE-VOTE RESP] peer 3 (O), term 0, resp term 0, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2 2024.09.25 20:43:44.523155 [ 18 ] {} RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 2024.09.25 20:43:44.523163 [ 18 ] {} RaftInstance: [PRE-VOTE DONE] initiate actual vote 2024.09.25 20:43:44.524605 [ 18 ] {} RaftInstance: [VOTE INIT] my id 1, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1) 2024.09.25 20:43:44.524637 [ 18 ] {} RaftInstance: send request_vote_request to server 3 with term 1 2024.09.25 20:43:44.524752 [ 18 ] {} RaftInstance: send req 1 -> 3, type request_vote_request 2024.09.25 20:43:44.524795 [ 18 ] {} RaftInstance: send request_vote_request to server 2 with term 1 2024.09.25 20:43:44.524807 [ 18 ] {} RaftInstance: send req 1 -> 2, type request_vote_request 2024.09.25 20:43:44.524831 [ 18 ] {} RaftInstance: cancel existing timer 2024.09.25 20:43:44.524850 [ 18 ] {} RaftInstance: re-schedule election timer 2024.09.25 20:43:44.524899 [ 12 ] {} RaftInstance: Receive a pre_vote_response message from peer 2 with Result=1, Term=0, NextIndex=1 2024.09.25 20:43:44.524917 [ 12 ] {} RaftInstance: src: 2, dst: 1, resp->get_term(): 0 2024.09.25 20:43:44.524928 [ 12 ] {} RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 0, resp term 0, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2 2024.09.25 20:43:44.524946 [ 12 ] {} RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 2024.09.25 20:43:44.524962 [ 12 ] {} RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing 2024.09.25 20:43:44.528135 [ 16 ] {} RaftInstance: resp of req 1 -> 3, type request_vote_request, OK 2024.09.25 20:43:44.528167 [ 16 ] {} RaftInstance: Receive a request_vote_response message from peer 3 with Result=1, Term=1, NextIndex=1 2024.09.25 20:43:44.528183 [ 16 ] {} RaftInstance: src: 3, dst: 1, resp->get_term(): 1 2024.09.25 20:43:44.528193 [ 16 ] {} RaftInstance: [VOTE RESP] peer 3 (O), resp term 1, my role candidate, granted 2, responded 2, num voting members 3, quorum 2 2024.09.25 20:43:44.528202 [ 16 ] {} RaftInstance: Server is elected as leader for term 1 2024.09.25 20:43:44.528227 [ 16 ] {} RaftInstance: number of pending commit elements: 0 2024.09.25 20:43:44.528236 [ 16 ] {} RaftInstance: state machine commit index 0, precommit index 0, last log index 0 2024.09.25 20:43:44.528249 [ 16 ] {} RaftInstance: peer 3, interval: 500 2024.09.25 20:43:44.528274 [ 16 ] {} RaftInstance: peer 2, interval: 500 2024.09.25 20:43:44.528309 [ 16 ] {} RaftInstance: [BECOME LEADER] appended new config at 1 2024.09.25 20:43:44.528536 [ 13 ] {} RaftInstance: resp of req 1 -> 2, type request_vote_request, OK 2024.09.25 20:43:44.529558 [ 16 ] {} RaftInstance: send request to 3 2024.09.25 20:43:44.529567 [ 9 ] {} RaftInstance: got log append completion notification: OK 2024.09.25 20:43:44.529593 [ 16 ] {} RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 2024.09.25 20:43:44.529637 [ 16 ] {} RaftInstance: append_entries for 3 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=1, peer_last_sent_idx 0 2024.09.25 20:43:44.529647 [ 16 ] {} RaftInstance: idx: 1 2024.09.25 20:43:44.529657 [ 16 ] {} RaftInstance: send req 1 -> 3, type append_entries_request 2024.09.25 20:43:44.529703 [ 16 ] {} RaftInstance: sent 2024.09.25 20:43:44.529715 [ 16 ] {} RaftInstance: send request to 2 2024.09.25 20:43:44.529723 [ 16 ] {} RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 2024.09.25 20:43:44.529733 [ 16 ] {} RaftInstance: append_entries for 2 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=1, peer_last_sent_idx 0 2024.09.25 20:43:44.529755 [ 16 ] {} RaftInstance: idx: 1 2024.09.25 20:43:44.529764 [ 16 ] {} RaftInstance: send req 1 -> 2, type append_entries_request 2024.09.25 20:43:44.529803 [ 16 ] {} RaftInstance: sent 2024.09.25 20:43:44.529812 [ 16 ] {} RaftInstance: === LEADER (term 1) === 2024.09.25 20:43:44.529875 [ 9 ] {} RaftInstance: last durable index 1, precommit index 1 2024.09.25 20:43:44.529893 [ 9 ] {} RaftInstance: quorum idx 1, 1 0 0 2024.09.25 20:43:44.529915 [ 9 ] {} RaftInstance: local log idx 1, target_commit_idx 0, quick_commit_index_ 0, state_->get_commit_idx() 0 2024.09.25 20:43:44.529963 [ 13 ] {} RaftInstance: Receive a request_vote_response message from peer 2 with Result=1, Term=1, NextIndex=1 2024.09.25 20:43:44.530028 [ 13 ] {} RaftInstance: src: 2, dst: 1, resp->get_term(): 1 2024.09.25 20:43:44.530072 [ 13 ] {} RaftInstance: Election completed, will ignore the voting result from this server 2024.09.25 20:43:44.531673 [ 12 ] {} RaftInstance: resp of req 1 -> 3, type append_entries_request, OK 2024.09.25 20:43:44.531702 [ 12 ] {} RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2 2024.09.25 20:43:44.531721 [ 12 ] {} RaftInstance: src: 3, dst: 1, resp->get_term(): 1 2024.09.25 20:43:44.531731 [ 12 ] {} RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 2024.09.25 20:43:44.531739 [ 12 ] {} RaftInstance: peer 3 batch size hint: 0 bytes 2024.09.25 20:43:44.531753 [ 12 ] {} RaftInstance: peer 3, prev matched idx: 0, new matched idx: 1 2024.09.25 20:43:44.531771 [ 12 ] {} RaftInstance: last durable index 1, precommit index 1 2024.09.25 20:43:44.531782 [ 12 ] {} RaftInstance: quorum idx 1, 1 1 0 2024.09.25 20:43:44.531791 [ 12 ] {} RaftInstance: trigger commit upto 1 2024.09.25 20:43:44.531801 [ 12 ] {} RaftInstance: send request to 3 2024.09.25 20:43:44.531793 [ 13 ] {} RaftInstance: resp of req 1 -> 2, type append_entries_request, OK 2024.09.25 20:43:44.531809 [ 12 ] {} RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 2024.09.25 20:43:44.531832 [ 12 ] {} RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 1 2024.09.25 20:43:44.531839 [ 12 ] {} RaftInstance: EMPTY PAYLOAD 2024.09.25 20:43:44.531847 [ 12 ] {} RaftInstance: send req 1 -> 3, type append_entries_request 2024.09.25 20:43:44.531892 [ 12 ] {} RaftInstance: sent 2024.09.25 20:43:44.531904 [ 12 ] {} RaftInstance: send request to 2 2024.09.25 20:43:44.531913 [ 12 ] {} RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 2024.09.25 20:43:44.531930 [ 12 ] {} RaftInstance: append_entries for 2 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1 2024.09.25 20:43:44.531940 [ 12 ] {} RaftInstance: idx: 1 2024.09.25 20:43:44.531952 [ 12 ] {} RaftInstance: send req 1 -> 2, type append_entries_request 2024.09.25 20:43:44.531969 [ 12 ] {} RaftInstance: sent 2024.09.25 20:43:44.531985 [ 12 ] {} RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0 2024.09.25 20:43:44.531995 [ 12 ] {} RaftInstance: commit_cv_ notify (local thread) 2024.09.25 20:43:44.532039 [ 20 ] {} RaftInstance: commit_cv_ wake up 2024.09.25 20:43:44.532066 [ 20 ] {} RaftInstance: commit upto 1, current idx 0 2024.09.25 20:43:44.532091 [ 20 ] {} RaftInstance: commit upto 1, current idx 1 2024.09.25 20:43:44.532104 [ 20 ] {} RaftInstance: config at index 1 is committed, prev config log idx 0 2024.09.25 20:43:44.532192 [ 20 ] {} RaftInstance: new config log idx 1, prev log idx 0, cur config log idx 0, prev log idx 0 2024.09.25 20:43:44.532224 [ 20 ] {} RaftInstance: system is reconfigured to have 3 servers, last config index: 0, this config index: 1 2024.09.25 20:43:44.532292 [ 20 ] {} RaftInstance: new configuration: log idx 1, prev log idx 0 peer 1, DC ID 0, ::1:39101, voting member, 1 peer 2, DC ID 0, ::1:39102, voting member, 1 peer 3, DC ID 0, ::1:39103, voting member, 1 my id: 1, leader: 1, term: 1 2024.09.25 20:43:44.532305 [ 20 ] {} RaftInstance: (update) new target priority: 1 2024.09.25 20:43:44.532330 [ 20 ] {} RaftInstance: DONE: commit upto 1, current idx 1 2024.09.25 20:43:44.532355 [ 20 ] {} RaftInstance: commit_cv_ sleep 2024.09.25 20:43:44.532387 [ 1 ] {} KeeperDispatcher: Quorum initialized 2024.09.25 20:43:44.532434 [ 18 ] {} RaftInstance: resp of req 1 -> 3, type append_entries_request, OK 2024.09.25 20:43:44.532461 [ 18 ] {} RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2 2024.09.25 20:43:44.532471 [ 16 ] {} RaftInstance: resp of req 1 -> 2, type append_entries_request, OK 2024.09.25 20:43:44.532473 [ 18 ] {} RaftInstance: src: 3, dst: 1, resp->get_term(): 1 2024.09.25 20:43:44.532512 [ 18 ] {} RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 2024.09.25 20:43:44.532522 [ 18 ] {} RaftInstance: peer 3 batch size hint: 0 bytes 2024.09.25 20:43:44.532524 [ 1 ] {} KeeperDispatcher: Dispatcher initialized 2024.09.25 20:43:44.532530 [ 18 ] {} RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1 2024.09.25 20:43:44.532560 [ 18 ] {} RaftInstance: last durable index 1, precommit index 1 2024.09.25 20:43:44.532573 [ 18 ] {} RaftInstance: quorum idx 1, 1 1 0 2024.09.25 20:43:44.532583 [ 18 ] {} RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 2024.09.25 20:43:44.532651 [ 13 ] {} RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=1, NextIndex=2 2024.09.25 20:43:44.532671 [ 13 ] {} RaftInstance: src: 2, dst: 1, resp->get_term(): 1 2024.09.25 20:43:44.532689 [ 13 ] {} RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 2024.09.25 20:43:44.532708 [ 13 ] {} RaftInstance: peer 2 batch size hint: 0 bytes 2024.09.25 20:43:44.532717 [ 13 ] {} RaftInstance: peer 2, prev matched idx: 0, new matched idx: 1 2024.09.25 20:43:44.532726 [ 13 ] {} RaftInstance: last durable index 1, precommit index 1 2024.09.25 20:43:44.532736 [ 13 ] {} RaftInstance: quorum idx 1, 1 1 1 2024.09.25 20:43:44.532760 [ 13 ] {} RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 2024.09.25 20:43:44.532803 [ 16 ] {} RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=1, NextIndex=2 2024.09.25 20:43:44.532825 [ 16 ] {} RaftInstance: src: 2, dst: 1, resp->get_term(): 1 2024.09.25 20:43:44.532835 [ 16 ] {} RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 2024.09.25 20:43:44.532934 [ 16 ] {} RaftInstance: peer 2 batch size hint: 0 bytes 2024.09.25 20:43:44.532952 [ 16 ] {} RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1 2024.09.25 20:43:44.532969 [ 16 ] {} RaftInstance: last durable index 1, precommit index 1 2024.09.25 20:43:44.532981 [ 16 ] {} RaftInstance: quorum idx 1, 1 1 1 2024.09.25 20:43:44.532990 [ 16 ] {} RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 2024.09.25 20:43:44.591993 [ 1 ] {} Context: Shutting down named sessions 2024.09.25 20:43:44.592069 [ 1 ] {} Context: Shutting down database catalog 2024.09.25 20:43:44.592090 [ 1 ] {} DatabaseCatalog: Shutting down system databases 2024.09.25 20:43:44.592117 [ 1 ] {} Context: Shutting down DDLWorker 2024.09.25 20:43:44.592126 [ 1 ] {} Context: Shutting down caches 2024.09.25 20:43:44.592239 [ 1 ] {} KeeperDispatcher: Shutting down storage dispatcher 2024.09.25 20:43:45.028656 [ 12 ] {} RaftInstance: heartbeat timeout for 3 2024.09.25 20:43:45.028691 [ 12 ] {} RaftInstance: (update) new target priority: 1 2024.09.25 20:43:45.028701 [ 12 ] {} RaftInstance: send request to 3 2024.09.25 20:43:45.028711 [ 12 ] {} RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 2024.09.25 20:43:45.028722 [ 12 ] {} RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2 2024.09.25 20:43:45.028753 [ 12 ] {} RaftInstance: EMPTY PAYLOAD 2024.09.25 20:43:45.028764 [ 12 ] {} RaftInstance: send req 1 -> 3, type append_entries_request 2024.09.25 20:43:45.028807 [ 12 ] {} RaftInstance: sent 2024.09.25 20:43:45.028872 [ 14 ] {} RaftInstance: heartbeat timeout for 2 2024.09.25 20:43:45.028891 [ 14 ] {} RaftInstance: (update) new target priority: 1 2024.09.25 20:43:45.028900 [ 14 ] {} RaftInstance: send request to 2 2024.09.25 20:43:45.028919 [ 14 ] {} RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 2024.09.25 20:43:45.028928 [ 14 ] {} RaftInstance: append_entries for 2 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 1 2024.09.25 20:43:45.028936 [ 14 ] {} RaftInstance: EMPTY PAYLOAD 2024.09.25 20:43:45.028945 [ 14 ] {} RaftInstance: send req 1 -> 2, type append_entries_request 2024.09.25 20:43:45.029017 [ 14 ] {} RaftInstance: sent 2024.09.25 20:43:45.029069 [ 16 ] {} RaftInstance: resp of req 1 -> 3, type append_entries_request, OK 2024.09.25 20:43:45.029098 [ 16 ] {} RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2 2024.09.25 20:43:45.029111 [ 16 ] {} RaftInstance: src: 3, dst: 1, resp->get_term(): 1 2024.09.25 20:43:45.029124 [ 16 ] {} RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 2024.09.25 20:43:45.029159 [ 16 ] {} RaftInstance: peer 3 batch size hint: 0 bytes 2024.09.25 20:43:45.029170 [ 16 ] {} RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1 2024.09.25 20:43:45.029182 [ 16 ] {} RaftInstance: last durable index 1, precommit index 1 2024.09.25 20:43:45.029194 [ 16 ] {} RaftInstance: quorum idx 1, 1 1 1 2024.09.25 20:43:45.029204 [ 16 ] {} RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 2024.09.25 20:43:45.029278 [ 16 ] {} RaftInstance: resp of req 1 -> 2, type append_entries_request, OK 2024.09.25 20:43:45.029299 [ 16 ] {} RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=1, NextIndex=2 2024.09.25 20:43:45.029309 [ 16 ] {} RaftInstance: src: 2, dst: 1, resp->get_term(): 1 2024.09.25 20:43:45.029318 [ 16 ] {} RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 2024.09.25 20:43:45.029333 [ 16 ] {} RaftInstance: peer 2 batch size hint: 0 bytes 2024.09.25 20:43:45.029342 [ 16 ] {} RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1 2024.09.25 20:43:45.029352 [ 16 ] {} RaftInstance: last durable index 1, precommit index 1 2024.09.25 20:43:45.029361 [ 16 ] {} RaftInstance: quorum idx 1, 1 1 1 2024.09.25 20:43:45.029371 [ 16 ] {} RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 2024.09.25 20:43:45.039734 [ 1 ] {} RaftInstance: shutting down raft core 2024.09.25 20:43:45.039782 [ 1 ] {} RaftInstance: sent stop signal to the commit thread. 2024.09.25 20:43:45.039814 [ 1 ] {} RaftInstance: asio client destroyed: 1d87d690 2024.09.25 20:43:45.039963 [ 20 ] {} RaftInstance: commit_cv_ wake up 2024.09.25 20:43:45.040880 [ 1 ] {} RaftInstance: asio client destroyed: 1d86ac30 2024.09.25 20:43:45.040985 [ 1 ] {} RaftInstance: cancelled all schedulers. 2024.09.25 20:43:45.041035 [ 1 ] {} RaftInstance: commit thread stopped. 2024.09.25 20:43:45.041047 [ 1 ] {} RaftInstance: all pending commit elements dropped. 2024.09.25 20:43:45.041056 [ 1 ] {} RaftInstance: reset all pointers. 2024.09.25 20:43:45.041124 [ 1 ] {} RaftInstance: joined terminated commit thread. 2024.09.25 20:43:45.041178 [ 21 ] {} RaftInstance: bg append_entries thread terminated 2024.09.25 20:43:45.041412 [ 1 ] {} RaftInstance: sent stop signal to background append thread. 2024.09.25 20:43:45.041441 [ 1 ] {} RaftInstance: clean up auto-forwarding queue: 0 elems 2024.09.25 20:43:45.041450 [ 1 ] {} RaftInstance: clean up auto-forwarding clients 2024.09.25 20:43:45.041463 [ 1 ] {} RaftInstance: raft_server shutdown completed. 2024.09.25 20:43:45.041471 [ 1 ] {} RaftInstance: manually create a snapshot on 1 2024.09.25 20:43:45.041480 [ 1 ] {} RaftInstance: creating a snapshot for index 1 2024.09.25 20:43:45.041501 [ 1 ] {} RaftInstance: create snapshot idx 1 log_term 1 2024.09.25 20:43:45.041517 [ 1 ] {} KeeperStateMachine: Creating snapshot 1 2024.09.25 20:43:45.041563 [ 1 ] {} KeeperStateMachine: Creating a snapshot during shutdown because 'create_snapshot_on_exit' is enabled. 2024.09.25 20:43:45.043732 [ 1 ] {} KeeperStateMachine: Created persistent snapshot 1 with path snapshot_1.bin.zstd 2024.09.25 20:43:45.043780 [ 1 ] {} KeeperStateMachine: Clearing garbage after snapshot 2024.09.25 20:43:45.043895 [ 1 ] {} KeeperStateMachine: Cleared garbage after snapshot 2024.09.25 20:43:45.043918 [ 1 ] {} RaftInstance: snapshot idx 1 log_term 1 created, compact the log store if needed 2024.09.25 20:43:45.043930 [ 1 ] {} KeeperStateMachine: Uploading snapshot snapshot_1.bin.zstd during shutdown because 'upload_snapshot_on_exit' is enabled. 2024.09.25 20:43:45.043966 [ 1 ] {} RaftInstance: create snapshot idx 1 log_term 1 done: 2453 us elapsed 2024.09.25 20:43:45.044039 [ 16 ] {} RaftInstance: failed to accept a rpc connection due to error 47, Operation canceled 2024.09.25 20:43:45.044104 [ 16 ] {} RaftInstance: asio rpc session destroyed: 1d86a580 2024.09.25 20:43:45.054287 [ 16 ] {} RaftInstance: end of asio worker thread, remaining threads: 7 2024.09.25 20:43:45.054324 [ 15 ] {} RaftInstance: end of asio worker thread, remaining threads: 6 2024.09.25 20:43:45.054360 [ 19 ] {} RaftInstance: end of asio worker thread, remaining threads: 4 2024.09.25 20:43:45.054389 [ 14 ] {} RaftInstance: end of asio worker thread, remaining threads: 0 2024.09.25 20:43:45.054378 [ 17 ] {} RaftInstance: end of asio worker thread, remaining threads: 3 2024.09.25 20:43:45.054381 [ 18 ] {} RaftInstance: end of asio worker thread, remaining threads: 1 2024.09.25 20:43:45.054381 [ 12 ] {} RaftInstance: end of asio worker thread, remaining threads: 1 2024.09.25 20:43:45.054318 [ 13 ] {} RaftInstance: end of asio worker thread, remaining threads: 5 2024.09.25 20:43:45.055806 [ 9 ] {} KeeperLogStore: Raft server is not set in LogStore. 2024.09.25 20:43:45.071210 [ 1 ] {} KeeperDispatcher: Dispatcher shut down 2024.09.25 20:43:45.072153 [ 1 ] {} KeeperLogStore: Changelog is shut down 2024.09.25 20:43:45.093563 [ 1 ] {} Application: Code: 210. DB::Exception: Listen [::1]:39001 failed: Poco::Exception. Code: 1000, e.code() = 125, Net Exception: Address already in use: [::1]:39001 (version 23.8.7.1). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 2. DB::Exception::Exception, std::allocator > const&, int&, std::__cxx11::basic_string, std::allocator > >(int, FormatStringHelperImpl, std::allocator > const&>::type, std::type_identity::type, std::type_identity, std::allocator > >::type>, std::__cxx11::basic_string, std::allocator > const&, int&, std::__cxx11::basic_string, std::allocator >&&) @ 0x000000001008a305 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 3. DB::Keeper::createServer(std::__cxx11::basic_string, std::allocator > const&, char const*, bool, std::function&&) const @ 0x000000001015f6cd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 4. DB::Keeper::main(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) @ 0x00000000101626c0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 5. Poco::Util::Application::run() @ 0x0000000019121669 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 6. DB::Keeper::run() @ 0x000000001015fb6b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 7. Poco::Util::ServerApplication::run(int, char**) @ 0x00000000191351a2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 8. mainEntryClickHouseKeeper(int, char**) @ 0x000000001015ef66 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 9. main @ 0x000000000aa294cc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse 10. _start_crt @ 0x000000000aa28bf7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse (version 23.8.7.1) 2024.09.25 20:43:45.093769 [ 1 ] {} Application: DB::Exception: Listen [::1]:39001 failed: Poco::Exception. Code: 1000, e.code() = 125, Net Exception: Address already in use: [::1]:39001 (version 23.8.7.1) 2024.09.25 20:43:45.093805 [ 1 ] {} Application: shutting down 2024.09.25 20:43:45.093822 [ 1 ] {} Application: Uninitializing subsystem: Logging Subsystem 2024.09.25 20:43:45.093897 [ 2 ] {} BaseDaemon: Received signal -2 2024.09.25 20:43:45.093929 [ 2 ] {} BaseDaemon: Stop SignalListener thread