| 1 | 2024.09.25 20:43:43.317264 [ 1 ] {} <Information> : Starting ClickHouse Keeper 23.8.7.1(revision : 54477, git hash: b6a391579f09dde47a1aa5602e154e7fcde80d1e, build id: <unknown>), PID 10493 | ||
| 2 | 2024.09.25 20:43:43.318919 [ 1 ] {} <Information> Application: starting up | ||
| 3 | 2024.09.25 20:43:43.319057 [ 1 ] {} <Information> Application: OS Name = SunOS, OS Version = 5.11, OS Architecture = i86pc | ||
| 4 | 2024.09.25 20:43:43.341329 [ 1 ] {} <Debug> Application: Initializing DateLUT. | ||
| 5 | 2024.09.25 20:43:43.341377 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'. | ||
| 6 | 2024.09.25 20:43:43.397565 [ 1 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously | ||
| 7 | 2024.09.25 20:43:43.397692 [ 1 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher | ||
| 8 | 2024.09.25 20:43:43.398380 [ 1 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled | ||
| 9 | 2024.09.25 20:43:43.398403 [ 1 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled | ||
| 10 | 2024.09.25 20:43:43.398430 [ 1 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled | ||
| 11 | 2024.09.25 20:43:43.398769 [ 1 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk | ||
| 12 | 2024.09.25 20:43:43.399209 [ 1 ] {} <Trace> KeeperLogStore: Reading from disk LocalLogDisk | ||
| 13 | 2024.09.25 20:43:43.399296 [ 1 ] {} <Warning> KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/integration_test-13fe08fd3bd99210-test_raft_config_parsing.10491.1-oximeter-clickward-test/keeper-1/coordination/log. It's Ok if it's the first run of clickhouse-keeper. | ||
| 14 | 2024.09.25 20:43:43.399540 [ 1 ] {} <Information> KeeperLogStore: force_sync enabled | ||
| 15 | 2024.09.25 20:43:43.399559 [ 1 ] {} <Debug> KeeperDispatcher: Waiting server to initialize | ||
| 16 | 2024.09.25 20:43:43.399579 [ 1 ] {} <Debug> KeeperStateMachine: Totally have 0 snapshots | ||
| 17 | 2024.09.25 20:43:43.399588 [ 1 ] {} <Debug> KeeperStateMachine: No existing snapshots, last committed log index 0 | ||
| 18 | 2024.09.25 20:43:43.399690 [ 1 ] {} <Warning> KeeperLogStore: Removing all changelogs | ||
| 19 | 2024.09.25 20:43:43.399823 [ 1 ] {} <Trace> Changelog: Starting new changelog changelog_1_100000.bin.zstd | ||
| 20 | 2024.09.25 20:43:43.399947 [ 1 ] {} <Information> KeeperServer: Preprocessing 0 log entries | ||
| 21 | 2024.09.25 20:43:43.399969 [ 1 ] {} <Information> KeeperServer: Preprocessing done | ||
| 22 | 2024.09.25 20:43:43.399979 [ 1 ] {} <Information> KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk | ||
| 23 | 2024.09.25 20:43:43.401678 [ 1 ] {} <Information> RaftInstance: Raft ASIO listener initiated on :::39101, unsecured | ||
| 24 | 2024.09.25 20:43:43.401841 [ 1 ] {} <Warning> KeeperStateManager: No state was read | ||
| 25 | 2024.09.25 20:43:43.401895 [ 1 ] {} <Information> RaftInstance: parameters: election timeout range 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 100000, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on | ||
| 26 | 2024.09.25 20:43:43.401914 [ 1 ] {} <Information> RaftInstance: new election timeout range: 1000 - 2000 | ||
| 27 | 2024.09.25 20:43:43.401990 [ 1 ] {} <Information> RaftInstance: === INIT RAFT SERVER === | ||
| 28 | commit index 0 | ||
| 29 | term 0 | ||
| 30 | election timer allowed | ||
| 31 | log store start 1, end 0 | ||
| 32 | config log idx 0, prev log idx 0 | ||
| 33 | 2024.09.25 20:43:43.402237 [ 1 ] {} <Trace> RaftInstance: asio client created: 1d86ac30 | ||
| 34 | 2024.09.25 20:43:43.402280 [ 1 ] {} <Trace> RaftInstance: asio client created: 1d87d690 | ||
| 35 | 2024.09.25 20:43:43.402293 [ 1 ] {} <Information> RaftInstance: peer 1: DC ID 0, ::1:39101, voting member, 1 | ||
| 36 | peer 2: DC ID 0, ::1:39102, voting member, 1 | ||
| 37 | peer 3: DC ID 0, ::1:39103, voting member, 1 | ||
| 38 | my id: 1, voting_member | ||
| 39 | num peers: 2 | ||
| 40 | 2024.09.25 20:43:43.402376 [ 1 ] {} <Information> RaftInstance: global manager does not exist. will use local thread for commit and append | ||
| 41 | 2024.09.25 20:43:43.402510 [ 1 ] {} <Information> RaftInstance: wait for HB, for 50 + [1000, 2000] ms | ||
| 42 | 2024.09.25 20:43:43.403023 [ 21 ] {} <Information> RaftInstance: bg append_entries thread initiated | ||
| 43 | 2024.09.25 20:43:43.454491 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep | ||
| 44 | 2024.09.25 20:43:43.460954 [ 1 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 45 | 2024.09.25 20:43:43.460997 [ 1 ] {} <Debug> RaftInstance: server 1 started | ||
| 46 | 2024.09.25 20:43:43.461040 [ 1 ] {} <Trace> RaftInstance: asio rpc session created: 1d86a580 | ||
| 47 | 2024.09.25 20:43:43.461148 [ 1 ] {} <Debug> KeeperDispatcher: Server initialized, waiting for quorum | ||
| 48 | 2024.09.25 20:43:44.521364 [ 13 ] {} <Trace> RaftInstance: election timeout | ||
| 49 | 2024.09.25 20:43:44.521426 [ 13 ] {} <Warning> RaftInstance: Election timeout, initiate leader election | ||
| 50 | 2024.09.25 20:43:44.521481 [ 13 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1 | ||
| 51 | 2024.09.25 20:43:44.521517 [ 13 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done | ||
| 52 | 2024.09.25 20:43:44.521589 [ 13 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 1, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 53 | 2024.09.25 20:43:44.521633 [ 13 ] {} <Trace> RaftInstance: send req 1 -> 3, type pre_vote_request | ||
| 54 | 2024.09.25 20:43:44.521662 [ 13 ] {} <Debug> RaftInstance: socket 1d87d690 to ::1:39103 is not opened yet | ||
| 55 | 2024.09.25 20:43:44.521800 [ 13 ] {} <Trace> RaftInstance: send req 1 -> 2, type pre_vote_request | ||
| 56 | 2024.09.25 20:43:44.521835 [ 13 ] {} <Debug> RaftInstance: socket 1d86ac30 to ::1:39102 is not opened yet | ||
| 57 | 2024.09.25 20:43:44.521849 [ 13 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 58 | 2024.09.25 20:43:44.521859 [ 13 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 59 | 2024.09.25 20:43:44.522553 [ 18 ] {} <Information> RaftInstance: 1d87d690 connected to ::1:39103 (as a client) | ||
| 60 | 2024.09.25 20:43:44.522689 [ 18 ] {} <Information> RaftInstance: 1d86ac30 connected to ::1:39102 (as a client) | ||
| 61 | 2024.09.25 20:43:44.523073 [ 18 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type pre_vote_request, OK | ||
| 62 | 2024.09.25 20:43:44.523100 [ 18 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 3 with Result=1, Term=0, NextIndex=1 | ||
| 63 | 2024.09.25 20:43:44.523102 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type pre_vote_request, OK | ||
| 64 | 2024.09.25 20:43:44.523110 [ 18 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 0 | ||
| 65 | 2024.09.25 20:43:44.523133 [ 18 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 3 (O), term 0, resp term 0, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 66 | 2024.09.25 20:43:44.523155 [ 18 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 | ||
| 67 | 2024.09.25 20:43:44.523163 [ 18 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote | ||
| 68 | 2024.09.25 20:43:44.524605 [ 18 ] {} <Information> RaftInstance: [VOTE INIT] my id 1, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 69 | 2024.09.25 20:43:44.524637 [ 18 ] {} <Debug> RaftInstance: send request_vote_request to server 3 with term 1 | ||
| 70 | 2024.09.25 20:43:44.524752 [ 18 ] {} <Trace> RaftInstance: send req 1 -> 3, type request_vote_request | ||
| 71 | 2024.09.25 20:43:44.524795 [ 18 ] {} <Debug> RaftInstance: send request_vote_request to server 2 with term 1 | ||
| 72 | 2024.09.25 20:43:44.524807 [ 18 ] {} <Trace> RaftInstance: send req 1 -> 2, type request_vote_request | ||
| 73 | 2024.09.25 20:43:44.524831 [ 18 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 74 | 2024.09.25 20:43:44.524850 [ 18 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 75 | 2024.09.25 20:43:44.524899 [ 12 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 2 with Result=1, Term=0, NextIndex=1 | ||
| 76 | 2024.09.25 20:43:44.524917 [ 12 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 0 | ||
| 77 | 2024.09.25 20:43:44.524928 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 0, resp term 0, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 78 | 2024.09.25 20:43:44.524946 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 | ||
| 79 | 2024.09.25 20:43:44.524962 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing | ||
| 80 | 2024.09.25 20:43:44.528135 [ 16 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type request_vote_request, OK | ||
| 81 | 2024.09.25 20:43:44.528167 [ 16 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 3 with Result=1, Term=1, NextIndex=1 | ||
| 82 | 2024.09.25 20:43:44.528183 [ 16 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 1 | ||
| 83 | 2024.09.25 20:43:44.528193 [ 16 ] {} <Information> RaftInstance: [VOTE RESP] peer 3 (O), resp term 1, my role candidate, granted 2, responded 2, num voting members 3, quorum 2 | ||
| 84 | 2024.09.25 20:43:44.528202 [ 16 ] {} <Information> RaftInstance: Server is elected as leader for term 1 | ||
| 85 | 2024.09.25 20:43:44.528227 [ 16 ] {} <Information> RaftInstance: number of pending commit elements: 0 | ||
| 86 | 2024.09.25 20:43:44.528236 [ 16 ] {} <Information> RaftInstance: state machine commit index 0, precommit index 0, last log index 0 | ||
| 87 | 2024.09.25 20:43:44.528249 [ 16 ] {} <Trace> RaftInstance: peer 3, interval: 500 | ||
| 88 | 2024.09.25 20:43:44.528274 [ 16 ] {} <Trace> RaftInstance: peer 2, interval: 500 | ||
| 89 | 2024.09.25 20:43:44.528309 [ 16 ] {} <Information> RaftInstance: [BECOME LEADER] appended new config at 1 | ||
| 90 | 2024.09.25 20:43:44.528536 [ 13 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type request_vote_request, OK | ||
| 91 | 2024.09.25 20:43:44.529558 [ 16 ] {} <Trace> RaftInstance: send request to 3 | ||
| 92 | 2024.09.25 20:43:44.529567 [ 9 ] {} <Trace> RaftInstance: got log append completion notification: OK | ||
| 93 | 2024.09.25 20:43:44.529593 [ 16 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 | ||
| 94 | 2024.09.25 20:43:44.529637 [ 16 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=1, peer_last_sent_idx 0 | ||
| 95 | 2024.09.25 20:43:44.529647 [ 16 ] {} <Debug> RaftInstance: idx: 1 | ||
| 96 | 2024.09.25 20:43:44.529657 [ 16 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request | ||
| 97 | 2024.09.25 20:43:44.529703 [ 16 ] {} <Trace> RaftInstance: sent | ||
| 98 | 2024.09.25 20:43:44.529715 [ 16 ] {} <Trace> RaftInstance: send request to 2 | ||
| 99 | 2024.09.25 20:43:44.529723 [ 16 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 | ||
| 100 | 2024.09.25 20:43:44.529733 [ 16 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=1, peer_last_sent_idx 0 | ||
| 101 | 2024.09.25 20:43:44.529755 [ 16 ] {} <Debug> RaftInstance: idx: 1 | ||
| 102 | 2024.09.25 20:43:44.529764 [ 16 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request | ||
| 103 | 2024.09.25 20:43:44.529803 [ 16 ] {} <Trace> RaftInstance: sent | ||
| 104 | 2024.09.25 20:43:44.529812 [ 16 ] {} <Information> RaftInstance: === LEADER (term 1) === | ||
| 105 | 2024.09.25 20:43:44.529875 [ 9 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 106 | 2024.09.25 20:43:44.529893 [ 9 ] {} <Trace> RaftInstance: quorum idx 1, 1 0 0 | ||
| 107 | 2024.09.25 20:43:44.529915 [ 9 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 0, quick_commit_index_ 0, state_->get_commit_idx() 0 | ||
| 108 | 2024.09.25 20:43:44.529963 [ 13 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 2 with Result=1, Term=1, NextIndex=1 | ||
| 109 | 2024.09.25 20:43:44.530028 [ 13 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1 | ||
| 110 | 2024.09.25 20:43:44.530072 [ 13 ] {} <Information> RaftInstance: Election completed, will ignore the voting result from this server | ||
| 111 | 2024.09.25 20:43:44.531673 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK | ||
| 112 | 2024.09.25 20:43:44.531702 [ 12 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2 | ||
| 113 | 2024.09.25 20:43:44.531721 [ 12 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 1 | ||
| 114 | 2024.09.25 20:43:44.531731 [ 12 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 | ||
| 115 | 2024.09.25 20:43:44.531739 [ 12 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes | ||
| 116 | 2024.09.25 20:43:44.531753 [ 12 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 0, new matched idx: 1 | ||
| 117 | 2024.09.25 20:43:44.531771 [ 12 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 118 | 2024.09.25 20:43:44.531782 [ 12 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 | ||
| 119 | 2024.09.25 20:43:44.531791 [ 12 ] {} <Debug> RaftInstance: trigger commit upto 1 | ||
| 120 | 2024.09.25 20:43:44.531801 [ 12 ] {} <Trace> RaftInstance: send request to 3 | ||
| 121 | 2024.09.25 20:43:44.531793 [ 13 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK | ||
| 122 | 2024.09.25 20:43:44.531809 [ 12 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 | ||
| 123 | 2024.09.25 20:43:44.531832 [ 12 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 1 | ||
| 124 | 2024.09.25 20:43:44.531839 [ 12 ] {} <Trace> RaftInstance: EMPTY PAYLOAD | ||
| 125 | 2024.09.25 20:43:44.531847 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request | ||
| 126 | 2024.09.25 20:43:44.531892 [ 12 ] {} <Trace> RaftInstance: sent | ||
| 127 | 2024.09.25 20:43:44.531904 [ 12 ] {} <Trace> RaftInstance: send request to 2 | ||
| 128 | 2024.09.25 20:43:44.531913 [ 12 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 | ||
| 129 | 2024.09.25 20:43:44.531930 [ 12 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=1, Term=1, peer_last_sent_idx 1 | ||
| 130 | 2024.09.25 20:43:44.531940 [ 12 ] {} <Debug> RaftInstance: idx: 1 | ||
| 131 | 2024.09.25 20:43:44.531952 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request | ||
| 132 | 2024.09.25 20:43:44.531969 [ 12 ] {} <Trace> RaftInstance: sent | ||
| 133 | 2024.09.25 20:43:44.531985 [ 12 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0 | ||
| 134 | 2024.09.25 20:43:44.531995 [ 12 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread) | ||
| 135 | 2024.09.25 20:43:44.532039 [ 20 ] {} <Trace> RaftInstance: commit_cv_ wake up | ||
| 136 | 2024.09.25 20:43:44.532066 [ 20 ] {} <Debug> RaftInstance: commit upto 1, current idx 0 | ||
| 137 | 2024.09.25 20:43:44.532091 [ 20 ] {} <Trace> RaftInstance: commit upto 1, current idx 1 | ||
| 138 | 2024.09.25 20:43:44.532104 [ 20 ] {} <Information> RaftInstance: config at index 1 is committed, prev config log idx 0 | ||
| 139 | 2024.09.25 20:43:44.532192 [ 20 ] {} <Information> RaftInstance: new config log idx 1, prev log idx 0, cur config log idx 0, prev log idx 0 | ||
| 140 | 2024.09.25 20:43:44.532224 [ 20 ] {} <Debug> RaftInstance: system is reconfigured to have 3 servers, last config index: 0, this config index: 1 | ||
| 141 | 2024.09.25 20:43:44.532292 [ 20 ] {} <Information> RaftInstance: new configuration: log idx 1, prev log idx 0 | ||
| 142 | peer 1, DC ID 0, ::1:39101, voting member, 1 | ||
| 143 | peer 2, DC ID 0, ::1:39102, voting member, 1 | ||
| 144 | peer 3, DC ID 0, ::1:39103, voting member, 1 | ||
| 145 | my id: 1, leader: 1, term: 1 | ||
| 146 | 2024.09.25 20:43:44.532305 [ 20 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 147 | 2024.09.25 20:43:44.532330 [ 20 ] {} <Debug> RaftInstance: DONE: commit upto 1, current idx 1 | ||
| 148 | 2024.09.25 20:43:44.532355 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep | ||
| 149 | 2024.09.25 20:43:44.532387 [ 1 ] {} <Debug> KeeperDispatcher: Quorum initialized | ||
| 150 | 2024.09.25 20:43:44.532434 [ 18 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK | ||
| 151 | 2024.09.25 20:43:44.532461 [ 18 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2 | ||
| 152 | 2024.09.25 20:43:44.532471 [ 16 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK | ||
| 153 | 2024.09.25 20:43:44.532473 [ 18 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 1 | ||
| 154 | 2024.09.25 20:43:44.532512 [ 18 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 | ||
| 155 | 2024.09.25 20:43:44.532522 [ 18 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes | ||
| 156 | 2024.09.25 20:43:44.532524 [ 1 ] {} <Debug> KeeperDispatcher: Dispatcher initialized | ||
| 157 | 2024.09.25 20:43:44.532530 [ 18 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1 | ||
| 158 | 2024.09.25 20:43:44.532560 [ 18 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 159 | 2024.09.25 20:43:44.532573 [ 18 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 | ||
| 160 | 2024.09.25 20:43:44.532583 [ 18 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 161 | 2024.09.25 20:43:44.532651 [ 13 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=1, NextIndex=2 | ||
| 162 | 2024.09.25 20:43:44.532671 [ 13 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1 | ||
| 163 | 2024.09.25 20:43:44.532689 [ 13 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 | ||
| 164 | 2024.09.25 20:43:44.532708 [ 13 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes | ||
| 165 | 2024.09.25 20:43:44.532717 [ 13 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 0, new matched idx: 1 | ||
| 166 | 2024.09.25 20:43:44.532726 [ 13 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 167 | 2024.09.25 20:43:44.532736 [ 13 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 168 | 2024.09.25 20:43:44.532760 [ 13 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 169 | 2024.09.25 20:43:44.532803 [ 16 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=1, NextIndex=2 | ||
| 170 | 2024.09.25 20:43:44.532825 [ 16 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1 | ||
| 171 | 2024.09.25 20:43:44.532835 [ 16 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 | ||
| 172 | 2024.09.25 20:43:44.532934 [ 16 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes | ||
| 173 | 2024.09.25 20:43:44.532952 [ 16 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1 | ||
| 174 | 2024.09.25 20:43:44.532969 [ 16 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 175 | 2024.09.25 20:43:44.532981 [ 16 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 176 | 2024.09.25 20:43:44.532990 [ 16 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 177 | 2024.09.25 20:43:44.591993 [ 1 ] {} <Trace> Context: Shutting down named sessions | ||
| 178 | 2024.09.25 20:43:44.592069 [ 1 ] {} <Trace> Context: Shutting down database catalog | ||
| 179 | 2024.09.25 20:43:44.592090 [ 1 ] {} <Trace> DatabaseCatalog: Shutting down system databases | ||
| 180 | 2024.09.25 20:43:44.592117 [ 1 ] {} <Trace> Context: Shutting down DDLWorker | ||
| 181 | 2024.09.25 20:43:44.592126 [ 1 ] {} <Trace> Context: Shutting down caches | ||
| 182 | 2024.09.25 20:43:44.592239 [ 1 ] {} <Debug> KeeperDispatcher: Shutting down storage dispatcher | ||
| 183 | 2024.09.25 20:43:45.028656 [ 12 ] {} <Debug> RaftInstance: heartbeat timeout for 3 | ||
| 184 | 2024.09.25 20:43:45.028691 [ 12 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 185 | 2024.09.25 20:43:45.028701 [ 12 ] {} <Trace> RaftInstance: send request to 3 | ||
| 186 | 2024.09.25 20:43:45.028711 [ 12 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 | ||
| 187 | 2024.09.25 20:43:45.028722 [ 12 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 2 | ||
| 188 | 2024.09.25 20:43:45.028753 [ 12 ] {} <Trace> RaftInstance: EMPTY PAYLOAD | ||
| 189 | 2024.09.25 20:43:45.028764 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request | ||
| 190 | 2024.09.25 20:43:45.028807 [ 12 ] {} <Trace> RaftInstance: sent | ||
| 191 | 2024.09.25 20:43:45.028872 [ 14 ] {} <Debug> RaftInstance: heartbeat timeout for 2 | ||
| 192 | 2024.09.25 20:43:45.028891 [ 14 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 193 | 2024.09.25 20:43:45.028900 [ 14 ] {} <Trace> RaftInstance: send request to 2 | ||
| 194 | 2024.09.25 20:43:45.028919 [ 14 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 | ||
| 195 | 2024.09.25 20:43:45.028928 [ 14 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=1, LastLogTerm=1, EntriesLength=0, CommitIndex=1, Term=1, peer_last_sent_idx 1 | ||
| 196 | 2024.09.25 20:43:45.028936 [ 14 ] {} <Trace> RaftInstance: EMPTY PAYLOAD | ||
| 197 | 2024.09.25 20:43:45.028945 [ 14 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request | ||
| 198 | 2024.09.25 20:43:45.029017 [ 14 ] {} <Trace> RaftInstance: sent | ||
| 199 | 2024.09.25 20:43:45.029069 [ 16 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK | ||
| 200 | 2024.09.25 20:43:45.029098 [ 16 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=1, NextIndex=2 | ||
| 201 | 2024.09.25 20:43:45.029111 [ 16 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 1 | ||
| 202 | 2024.09.25 20:43:45.029124 [ 16 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 | ||
| 203 | 2024.09.25 20:43:45.029159 [ 16 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes | ||
| 204 | 2024.09.25 20:43:45.029170 [ 16 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1 | ||
| 205 | 2024.09.25 20:43:45.029182 [ 16 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 206 | 2024.09.25 20:43:45.029194 [ 16 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 207 | 2024.09.25 20:43:45.029204 [ 16 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 208 | 2024.09.25 20:43:45.029278 [ 16 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK | ||
| 209 | 2024.09.25 20:43:45.029299 [ 16 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=1, NextIndex=2 | ||
| 210 | 2024.09.25 20:43:45.029309 [ 16 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1 | ||
| 211 | 2024.09.25 20:43:45.029318 [ 16 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 | ||
| 212 | 2024.09.25 20:43:45.029333 [ 16 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes | ||
| 213 | 2024.09.25 20:43:45.029342 [ 16 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1 | ||
| 214 | 2024.09.25 20:43:45.029352 [ 16 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 215 | 2024.09.25 20:43:45.029361 [ 16 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 216 | 2024.09.25 20:43:45.029371 [ 16 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 217 | 2024.09.25 20:43:45.039734 [ 1 ] {} <Information> RaftInstance: shutting down raft core | ||
| 218 | 2024.09.25 20:43:45.039782 [ 1 ] {} <Information> RaftInstance: sent stop signal to the commit thread. | ||
| 219 | 2024.09.25 20:43:45.039814 [ 1 ] {} <Trace> RaftInstance: asio client destroyed: 1d87d690 | ||
| 220 | 2024.09.25 20:43:45.039963 [ 20 ] {} <Trace> RaftInstance: commit_cv_ wake up | ||
| 221 | 2024.09.25 20:43:45.040880 [ 1 ] {} <Trace> RaftInstance: asio client destroyed: 1d86ac30 | ||
| 222 | 2024.09.25 20:43:45.040985 [ 1 ] {} <Information> RaftInstance: cancelled all schedulers. | ||
| 223 | 2024.09.25 20:43:45.041035 [ 1 ] {} <Information> RaftInstance: commit thread stopped. | ||
| 224 | 2024.09.25 20:43:45.041047 [ 1 ] {} <Information> RaftInstance: all pending commit elements dropped. | ||
| 225 | 2024.09.25 20:43:45.041056 [ 1 ] {} <Information> RaftInstance: reset all pointers. | ||
| 226 | 2024.09.25 20:43:45.041124 [ 1 ] {} <Information> RaftInstance: joined terminated commit thread. | ||
| 227 | 2024.09.25 20:43:45.041178 [ 21 ] {} <Information> RaftInstance: bg append_entries thread terminated | ||
| 228 | 2024.09.25 20:43:45.041412 [ 1 ] {} <Information> RaftInstance: sent stop signal to background append thread. | ||
| 229 | 2024.09.25 20:43:45.041441 [ 1 ] {} <Information> RaftInstance: clean up auto-forwarding queue: 0 elems | ||
| 230 | 2024.09.25 20:43:45.041450 [ 1 ] {} <Information> RaftInstance: clean up auto-forwarding clients | ||
| 231 | 2024.09.25 20:43:45.041463 [ 1 ] {} <Information> RaftInstance: raft_server shutdown completed. | ||
| 232 | 2024.09.25 20:43:45.041471 [ 1 ] {} <Information> RaftInstance: manually create a snapshot on 1 | ||
| 233 | 2024.09.25 20:43:45.041480 [ 1 ] {} <Information> RaftInstance: creating a snapshot for index 1 | ||
| 234 | 2024.09.25 20:43:45.041501 [ 1 ] {} <Information> RaftInstance: create snapshot idx 1 log_term 1 | ||
| 235 | 2024.09.25 20:43:45.041517 [ 1 ] {} <Debug> KeeperStateMachine: Creating snapshot 1 | ||
| 236 | 2024.09.25 20:43:45.041563 [ 1 ] {} <Information> KeeperStateMachine: Creating a snapshot during shutdown because 'create_snapshot_on_exit' is enabled. | ||
| 237 | 2024.09.25 20:43:45.043732 [ 1 ] {} <Debug> KeeperStateMachine: Created persistent snapshot 1 with path snapshot_1.bin.zstd | ||
| 238 | 2024.09.25 20:43:45.043780 [ 1 ] {} <Trace> KeeperStateMachine: Clearing garbage after snapshot | ||
| 239 | 2024.09.25 20:43:45.043895 [ 1 ] {} <Trace> KeeperStateMachine: Cleared garbage after snapshot | ||
| 240 | 2024.09.25 20:43:45.043918 [ 1 ] {} <Information> RaftInstance: snapshot idx 1 log_term 1 created, compact the log store if needed | ||
| 241 | 2024.09.25 20:43:45.043930 [ 1 ] {} <Information> KeeperStateMachine: Uploading snapshot snapshot_1.bin.zstd during shutdown because 'upload_snapshot_on_exit' is enabled. | ||
| 242 | 2024.09.25 20:43:45.043966 [ 1 ] {} <Information> RaftInstance: create snapshot idx 1 log_term 1 done: 2453 us elapsed | ||
| 243 | 2024.09.25 20:43:45.044039 [ 16 ] {} <Error> RaftInstance: failed to accept a rpc connection due to error 47, Operation canceled | ||
| 244 | 2024.09.25 20:43:45.044104 [ 16 ] {} <Trace> RaftInstance: asio rpc session destroyed: 1d86a580 | ||
| 245 | 2024.09.25 20:43:45.054287 [ 16 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 7 | ||
| 246 | 2024.09.25 20:43:45.054324 [ 15 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 6 | ||
| 247 | 2024.09.25 20:43:45.054360 [ 19 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 4 | ||
| 248 | 2024.09.25 20:43:45.054389 [ 14 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 0 | ||
| 249 | 2024.09.25 20:43:45.054378 [ 17 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 3 | ||
| 250 | 2024.09.25 20:43:45.054381 [ 18 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 1 | ||
| 251 | 2024.09.25 20:43:45.054381 [ 12 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 1 | ||
| 252 | 2024.09.25 20:43:45.054318 [ 13 ] {} <Information> RaftInstance: end of asio worker thread, remaining threads: 5 | ||
| 253 | 2024.09.25 20:43:45.055806 [ 9 ] {} <Warning> KeeperLogStore: Raft server is not set in LogStore. | ||
| 254 | 2024.09.25 20:43:45.071210 [ 1 ] {} <Debug> KeeperDispatcher: Dispatcher shut down | ||
| 255 | 2024.09.25 20:43:45.072153 [ 1 ] {} <Warning> KeeperLogStore: Changelog is shut down | ||
| 256 | 2024.09.25 20:43:45.093563 [ 1 ] {} <Error> Application: Code: 210. DB::Exception: Listen [::1]:39001 failed: Poco::Exception. Code: 1000, e.code() = 125, Net Exception: Address already in use: [::1]:39001 (version 23.8.7.1). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): | ||
| 257 | |||
| 258 | 0. StackTrace::tryCapture() @ 0x0000000010014f57 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 259 | 1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ffe797f in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 260 | 2. DB::Exception::Exception<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, FormatStringHelperImpl<std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::type, std::type_identity<int&>::type, std::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000001008a305 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 261 | 3. DB::Keeper::createServer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, bool, std::function<void (unsigned short)>&&) const @ 0x000000001015f6cd in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 262 | 4. DB::Keeper::main(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) @ 0x00000000101626c0 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 263 | 5. Poco::Util::Application::run() @ 0x0000000019121669 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 264 | 6. DB::Keeper::run() @ 0x000000001015fb6b in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 265 | 7. Poco::Util::ServerApplication::run(int, char**) @ 0x00000000191351a2 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 266 | 8. mainEntryClickHouseKeeper(int, char**) @ 0x000000001015ef66 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 267 | 9. main @ 0x000000000aa294cc in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 268 | 10. _start_crt @ 0x000000000aa28bf7 in /work/oxidecomputer/omicron/out/clickhouse/clickhouse | ||
| 269 | (version 23.8.7.1) | ||
| 270 | 2024.09.25 20:43:45.093769 [ 1 ] {} <Error> Application: DB::Exception: Listen [::1]:39001 failed: Poco::Exception. Code: 1000, e.code() = 125, Net Exception: Address already in use: [::1]:39001 (version 23.8.7.1) | ||
| 271 | 2024.09.25 20:43:45.093805 [ 1 ] {} <Information> Application: shutting down | ||
| 272 | 2024.09.25 20:43:45.093822 [ 1 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem | ||
| 273 | 2024.09.25 20:43:45.093897 [ 2 ] {} <Trace> BaseDaemon: Received signal -2 | ||
| 274 | 2024.09.25 20:43:45.093929 [ 2 ] {} <Information> BaseDaemon: Stop SignalListener thread | ||