| 1 | 2025.05.14 01:41:21.494130 [ 1 ] {} <Information> : Starting ClickHouse Keeper 23.8.7.1(revision : 54477, git hash: 3042d295d963012962f5c683bd2776fa331a38c3, build id: <unknown>), PID 16476 | ||
| 2 | 2025.05.14 01:41:21.494370 [ 1 ] {} <Information> Application: starting up | ||
| 3 | 2025.05.14 01:41:21.494424 [ 1 ] {} <Information> Application: OS Name = SunOS, OS Version = 5.11, OS Architecture = i86pc | ||
| 4 | 2025.05.14 01:41:21.495977 [ 1 ] {} <Debug> Application: Initializing DateLUT. | ||
| 5 | 2025.05.14 01:41:21.496004 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'. | ||
| 6 | 2025.05.14 01:41:21.499704 [ 1 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously | ||
| 7 | 2025.05.14 01:41:21.499797 [ 1 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher | ||
| 8 | 2025.05.14 01:41:21.500302 [ 1 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled | ||
| 9 | 2025.05.14 01:41:21.500321 [ 1 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled | ||
| 10 | 2025.05.14 01:41:21.500343 [ 1 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled | ||
| 11 | 2025.05.14 01:41:21.500783 [ 1 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk | ||
| 12 | 2025.05.14 01:41:21.501281 [ 1 ] {} <Trace> KeeperLogStore: Reading from disk LocalLogDisk | ||
| 13 | 2025.05.14 01:41:21.501353 [ 1 ] {} <Warning> KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.1-clickhouse-iFcRob/log. It's Ok if it's the first run of clickhouse-keeper. | ||
| 14 | 2025.05.14 01:41:21.501563 [ 1 ] {} <Information> KeeperLogStore: force_sync enabled | ||
| 15 | 2025.05.14 01:41:21.501579 [ 1 ] {} <Debug> KeeperDispatcher: Waiting server to initialize | ||
| 16 | 2025.05.14 01:41:21.501588 [ 1 ] {} <Debug> KeeperStateMachine: Totally have 0 snapshots | ||
| 17 | 2025.05.14 01:41:21.501600 [ 1 ] {} <Debug> KeeperStateMachine: No existing snapshots, last committed log index 0 | ||
| 18 | 2025.05.14 01:41:21.501667 [ 1 ] {} <Warning> KeeperLogStore: Removing all changelogs | ||
| 19 | 2025.05.14 01:41:21.501759 [ 1 ] {} <Trace> Changelog: Starting new changelog changelog_1_100000.bin.zstd | ||
| 20 | 2025.05.14 01:41:21.501940 [ 1 ] {} <Information> KeeperServer: Preprocessing 0 log entries | ||
| 21 | 2025.05.14 01:41:21.501956 [ 1 ] {} <Information> KeeperServer: Preprocessing done | ||
| 22 | 2025.05.14 01:41:21.501986 [ 1 ] {} <Information> KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk | ||
| 23 | 2025.05.14 01:41:21.504664 [ 1 ] {} <Information> RaftInstance: Raft ASIO listener initiated on :::9234, unsecured | ||
| 24 | 2025.05.14 01:41:21.504942 [ 1 ] {} <Warning> KeeperStateManager: No state was read | ||
| 25 | 2025.05.14 01:41:21.505183 [ 1 ] {} <Information> RaftInstance: parameters: election timeout range 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 100000, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on | ||
| 26 | 2025.05.14 01:41:21.505246 [ 1 ] {} <Information> RaftInstance: new election timeout range: 1000 - 2000 | ||
| 27 | 2025.05.14 01:41:21.505312 [ 1 ] {} <Information> RaftInstance: === INIT RAFT SERVER === | ||
| 28 | commit index 0 | ||
| 29 | term 0 | ||
| 30 | election timer allowed | ||
| 31 | log store start 1, end 0 | ||
| 32 | config log idx 0, prev log idx 0 | ||
| 33 | 2025.05.14 01:41:21.505497 [ 1 ] {} <Trace> RaftInstance: asio client created: 1bc071e0 | ||
| 34 | 2025.05.14 01:41:21.505543 [ 1 ] {} <Trace> RaftInstance: asio client created: 1bc06060 | ||
| 35 | 2025.05.14 01:41:21.505560 [ 1 ] {} <Information> RaftInstance: peer 1: DC ID 0, ::1:9234, voting member, 1 | ||
| 36 | peer 2: DC ID 0, ::1:9235, voting member, 1 | ||
| 37 | peer 3: DC ID 0, ::1:9236, voting member, 1 | ||
| 38 | my id: 1, voting_member | ||
| 39 | num peers: 2 | ||
| 40 | 2025.05.14 01:41:21.505576 [ 1 ] {} <Information> RaftInstance: global manager does not exist. will use local thread for commit and append | ||
| 41 | 2025.05.14 01:41:21.505704 [ 1 ] {} <Information> RaftInstance: wait for HB, for 50 + [1000, 2000] ms | ||
| 42 | 2025.05.14 01:41:21.505779 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep | ||
| 43 | 2025.05.14 01:41:21.505830 [ 21 ] {} <Information> RaftInstance: bg append_entries thread initiated | ||
| 44 | 2025.05.14 01:41:21.556281 [ 1 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 45 | 2025.05.14 01:41:21.556340 [ 1 ] {} <Debug> RaftInstance: server 1 started | ||
| 46 | 2025.05.14 01:41:21.556414 [ 1 ] {} <Trace> RaftInstance: asio rpc session created: 1c9199a0 | ||
| 47 | 2025.05.14 01:41:21.556510 [ 1 ] {} <Debug> KeeperDispatcher: Server initialized, waiting for quorum | ||
| 48 | 2025.05.14 01:41:23.378180 [ 12 ] {} <Trace> RaftInstance: election timeout | ||
| 49 | 2025.05.14 01:41:23.378236 [ 12 ] {} <Warning> RaftInstance: Election timeout, initiate leader election | ||
| 50 | 2025.05.14 01:41:23.378286 [ 12 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1 | ||
| 51 | 2025.05.14 01:41:23.378303 [ 12 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done | ||
| 52 | 2025.05.14 01:41:23.378361 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 1, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 53 | 2025.05.14 01:41:23.378381 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type pre_vote_request | ||
| 54 | 2025.05.14 01:41:23.378392 [ 12 ] {} <Debug> RaftInstance: socket 1bc06060 to ::1:9236 is not opened yet | ||
| 55 | 2025.05.14 01:41:23.378575 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 2, type pre_vote_request | ||
| 56 | 2025.05.14 01:41:23.378596 [ 12 ] {} <Debug> RaftInstance: socket 1bc071e0 to ::1:9235 is not opened yet | ||
| 57 | 2025.05.14 01:41:23.378606 [ 12 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 58 | 2025.05.14 01:41:23.378612 [ 12 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 59 | 2025.05.14 01:41:23.379650 [ 19 ] {} <Information> RaftInstance: 1bc06060 connected to ::1:9236 (as a client) | ||
| 60 | 2025.05.14 01:41:23.379706 [ 12 ] {} <Information> RaftInstance: receive a incoming rpc connection | ||
| 61 | 2025.05.14 01:41:23.379769 [ 12 ] {} <Information> RaftInstance: session 1 got connection from ::1:41690 (as a server) | ||
| 62 | 2025.05.14 01:41:23.379788 [ 19 ] {} <Information> RaftInstance: 1bc071e0 connected to ::1:9235 (as a client) | ||
| 63 | 2025.05.14 01:41:23.379879 [ 12 ] {} <Trace> RaftInstance: asio rpc session created: 1c9190a0 | ||
| 64 | 2025.05.14 01:41:23.380250 [ 17 ] {} <Debug> RaftInstance: Receive a pre_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0 | ||
| 65 | 2025.05.14 01:41:23.380312 [ 17 ] {} <Information> RaftInstance: [PRE-VOTE REQ] my role candidate, from peer 2, log term: req 0 / mine 0 | ||
| 66 | last idx: req 0 / mine 0, term: req 0 / mine 0 | ||
| 67 | HB dead | ||
| 68 | 2025.05.14 01:41:23.380325 [ 17 ] {} <Information> RaftInstance: pre-vote decision: O (grant) | ||
| 69 | 2025.05.14 01:41:23.380337 [ 17 ] {} <Debug> RaftInstance: Response back a pre_vote_response message to 2 with Accepted=1, Term=0, NextIndex=1 | ||
| 70 | 2025.05.14 01:41:23.380553 [ 16 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type pre_vote_request, OK | ||
| 71 | 2025.05.14 01:41:23.380643 [ 16 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 2 with Result=1, Term=0, NextIndex=1 | ||
| 72 | 2025.05.14 01:41:23.380656 [ 16 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 0 | ||
| 73 | 2025.05.14 01:41:23.380670 [ 16 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 0, resp term 0, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 74 | 2025.05.14 01:41:23.380682 [ 16 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 | ||
| 75 | 2025.05.14 01:41:23.380694 [ 16 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote | ||
| 76 | 2025.05.14 01:41:23.380971 [ 16 ] {} <Information> RaftInstance: [VOTE INIT] my id 1, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 77 | 2025.05.14 01:41:23.380994 [ 16 ] {} <Debug> RaftInstance: send request_vote_request to server 3 with term 1 | ||
| 78 | 2025.05.14 01:41:23.380993 [ 17 ] {} <Debug> RaftInstance: Receive a request_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=1 | ||
| 79 | 2025.05.14 01:41:23.381012 [ 16 ] {} <Warning> RaftInstance: failed to send vote request: peer 3 (::1:9236) is busy | ||
| 80 | 2025.05.14 01:41:23.381051 [ 16 ] {} <Debug> RaftInstance: send request_vote_request to server 2 with term 1 | ||
| 81 | 2025.05.14 01:41:23.381057 [ 16 ] {} <Trace> RaftInstance: send req 1 -> 2, type request_vote_request | ||
| 82 | 2025.05.14 01:41:23.381123 [ 16 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 83 | 2025.05.14 01:41:23.381151 [ 16 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 84 | 2025.05.14 01:41:23.381291 [ 17 ] {} <Information> RaftInstance: [VOTE REQ] my role candidate, from peer 2, log term: req 0 / mine 0 | ||
| 85 | last idx: req 0 / mine 0, term: req 1 / mine 1 | ||
| 86 | priority: target 1 / mine 1, voted_for 1 | ||
| 87 | 2025.05.14 01:41:23.381301 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type request_vote_request, OK | ||
| 88 | 2025.05.14 01:41:23.381321 [ 17 ] {} <Information> RaftInstance: decision: X (deny), term 1 | ||
| 89 | 2025.05.14 01:41:23.381357 [ 17 ] {} <Debug> RaftInstance: Response back a request_vote_response message to 2 with Accepted=0, Term=1, NextIndex=0 | ||
| 90 | 2025.05.14 01:41:23.381367 [ 12 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 2 with Result=0, Term=1, NextIndex=0 | ||
| 91 | 2025.05.14 01:41:23.381366 [ 14 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type pre_vote_request, OK | ||
| 92 | 2025.05.14 01:41:23.381381 [ 12 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1 | ||
| 93 | 2025.05.14 01:41:23.381465 [ 12 ] {} <Information> RaftInstance: [VOTE RESP] peer 2 (X), resp term 1, my role candidate, granted 1, responded 2, num voting members 3, quorum 2 | ||
| 94 | 2025.05.14 01:41:23.381525 [ 14 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 3 with Result=1, Term=0, NextIndex=1 | ||
| 95 | 2025.05.14 01:41:23.381551 [ 14 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 0 | ||
| 96 | 2025.05.14 01:41:23.381564 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 3 (O), term 0, resp term 0, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 97 | 2025.05.14 01:41:23.381587 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 | ||
| 98 | 2025.05.14 01:41:23.381612 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing | ||
| 99 | 2025.05.14 01:41:24.333356 [ 17 ] {} <Information> RaftInstance: receive a incoming rpc connection | ||
| 100 | 2025.05.14 01:41:24.333420 [ 17 ] {} <Information> RaftInstance: session 2 got connection from ::1:64933 (as a server) | ||
| 101 | 2025.05.14 01:41:24.333533 [ 17 ] {} <Trace> RaftInstance: asio rpc session created: 1c94f6e0 | ||
| 102 | 2025.05.14 01:41:24.333573 [ 14 ] {} <Debug> RaftInstance: Receive a pre_vote_request message from 3 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0 | ||
| 103 | 2025.05.14 01:41:24.333637 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE REQ] my role candidate, from peer 3, log term: req 0 / mine 0 | ||
| 104 | last idx: req 0 / mine 0, term: req 0 / mine 1 | ||
| 105 | HB dead | ||
| 106 | 2025.05.14 01:41:24.333646 [ 14 ] {} <Information> RaftInstance: pre-vote decision: O (grant) | ||
| 107 | 2025.05.14 01:41:24.333669 [ 14 ] {} <Debug> RaftInstance: Response back a pre_vote_response message to 3 with Accepted=1, Term=0, NextIndex=1 | ||
| 108 | 2025.05.14 01:41:24.334350 [ 17 ] {} <Debug> RaftInstance: Receive a request_vote_request message from 3 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=1 | ||
| 109 | 2025.05.14 01:41:24.334392 [ 17 ] {} <Information> RaftInstance: [VOTE REQ] my role candidate, from peer 3, log term: req 0 / mine 0 | ||
| 110 | last idx: req 0 / mine 0, term: req 1 / mine 1 | ||
| 111 | priority: target 1 / mine 1, voted_for 1 | ||
| 112 | 2025.05.14 01:41:24.334415 [ 17 ] {} <Information> RaftInstance: decision: X (deny), term 1 | ||
| 113 | 2025.05.14 01:41:24.334424 [ 17 ] {} <Debug> RaftInstance: Response back a request_vote_response message to 3 with Accepted=0, Term=1, NextIndex=0 | ||
| 114 | 2025.05.14 01:41:24.777912 [ 14 ] {} <Trace> RaftInstance: election timeout | ||
| 115 | 2025.05.14 01:41:24.777991 [ 14 ] {} <Warning> RaftInstance: Election timeout, initiate leader election | ||
| 116 | 2025.05.14 01:41:24.778050 [ 14 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1 | ||
| 117 | 2025.05.14 01:41:24.778067 [ 14 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 1, target p 1, my p 1, hb dead, pre-vote done | ||
| 118 | 2025.05.14 01:41:24.778079 [ 14 ] {} <Information> RaftInstance: pre-vote term (0) is different, reset it to 1 | ||
| 119 | 2025.05.14 01:41:24.778100 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 1, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 120 | 2025.05.14 01:41:24.778121 [ 14 ] {} <Trace> RaftInstance: send req 1 -> 3, type pre_vote_request | ||
| 121 | 2025.05.14 01:41:24.778181 [ 14 ] {} <Trace> RaftInstance: send req 1 -> 2, type pre_vote_request | ||
| 122 | 2025.05.14 01:41:24.778266 [ 14 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 123 | 2025.05.14 01:41:24.778280 [ 14 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 124 | 2025.05.14 01:41:24.778459 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type pre_vote_request, OK | ||
| 125 | 2025.05.14 01:41:24.778526 [ 12 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 3 with Result=1, Term=1, NextIndex=1 | ||
| 126 | 2025.05.14 01:41:24.778534 [ 12 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 1 | ||
| 127 | 2025.05.14 01:41:24.778546 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 3 (O), term 1, resp term 1, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 128 | 2025.05.14 01:41:24.778557 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 1 | ||
| 129 | 2025.05.14 01:41:24.778566 [ 14 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type pre_vote_request, OK | ||
| 130 | 2025.05.14 01:41:24.778569 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote | ||
| 131 | 2025.05.14 01:41:24.778751 [ 12 ] {} <Debug> IDisk: Copying from LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.1-clickhouse-iFcRob/snapshots) state to LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.1-clickhouse-iFcRob/snapshots) state-OLD. | ||
| 132 | 2025.05.14 01:41:24.779153 [ 12 ] {} <Information> RaftInstance: [VOTE INIT] my id 1, my role candidate, term 2, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 133 | 2025.05.14 01:41:24.779176 [ 12 ] {} <Debug> RaftInstance: send request_vote_request to server 3 with term 2 | ||
| 134 | 2025.05.14 01:41:24.779184 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type request_vote_request | ||
| 135 | 2025.05.14 01:41:24.779250 [ 12 ] {} <Debug> RaftInstance: send request_vote_request to server 2 with term 2 | ||
| 136 | 2025.05.14 01:41:24.779258 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 2, type request_vote_request | ||
| 137 | 2025.05.14 01:41:24.779299 [ 12 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 138 | 2025.05.14 01:41:24.779322 [ 12 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 139 | 2025.05.14 01:41:24.779377 [ 14 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 2 with Result=1, Term=1, NextIndex=1 | ||
| 140 | 2025.05.14 01:41:24.779423 [ 14 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1 | ||
| 141 | 2025.05.14 01:41:24.779439 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 1, resp term 1, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 142 | 2025.05.14 01:41:24.779452 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 1 | ||
| 143 | 2025.05.14 01:41:24.779463 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing | ||
| 144 | 2025.05.14 01:41:24.780407 [ 19 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type request_vote_request, OK | ||
| 145 | 2025.05.14 01:41:24.780443 [ 19 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 2 with Result=1, Term=2, NextIndex=1 | ||
| 146 | 2025.05.14 01:41:24.780454 [ 19 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2 | ||
| 147 | 2025.05.14 01:41:24.780466 [ 19 ] {} <Information> RaftInstance: [VOTE RESP] peer 2 (O), resp term 2, my role candidate, granted 2, responded 2, num voting members 3, quorum 2 | ||
| 148 | 2025.05.14 01:41:24.780484 [ 19 ] {} <Information> RaftInstance: Server is elected as leader for term 2 | ||
| 149 | 2025.05.14 01:41:24.780518 [ 19 ] {} <Information> RaftInstance: number of pending commit elements: 0 | ||
| 150 | 2025.05.14 01:41:24.780543 [ 19 ] {} <Information> RaftInstance: state machine commit index 0, precommit index 0, last log index 0 | ||
| 151 | 2025.05.14 01:41:24.780555 [ 19 ] {} <Trace> RaftInstance: peer 3, interval: 500 | ||
| 152 | 2025.05.14 01:41:24.780567 [ 19 ] {} <Trace> RaftInstance: peer 2, interval: 500 | ||
| 153 | 2025.05.14 01:41:24.780574 [ 14 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type request_vote_request, OK | ||
| 154 | 2025.05.14 01:41:24.780589 [ 19 ] {} <Information> RaftInstance: [BECOME LEADER] appended new config at 1 | ||
| 155 | 2025.05.14 01:41:24.781027 [ 19 ] {} <Trace> RaftInstance: send request to 3 | ||
| 156 | 2025.05.14 01:41:24.781042 [ 10 ] {} <Trace> RaftInstance: got log append completion notification: OK | ||
| 157 | 2025.05.14 01:41:24.781052 [ 19 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 | ||
| 158 | 2025.05.14 01:41:24.781081 [ 19 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=2, peer_last_sent_idx 0 | ||
| 159 | 2025.05.14 01:41:24.781088 [ 19 ] {} <Debug> RaftInstance: idx: 1 | ||
| 160 | 2025.05.14 01:41:24.781100 [ 19 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request | ||
| 161 | 2025.05.14 01:41:24.781174 [ 19 ] {} <Trace> RaftInstance: sent | ||
| 162 | 2025.05.14 01:41:24.781183 [ 19 ] {} <Trace> RaftInstance: send request to 2 | ||
| 163 | 2025.05.14 01:41:24.781189 [ 19 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2 | ||
| 164 | 2025.05.14 01:41:24.781202 [ 19 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=2, peer_last_sent_idx 0 | ||
| 165 | 2025.05.14 01:41:24.781208 [ 19 ] {} <Debug> RaftInstance: idx: 1 | ||
| 166 | 2025.05.14 01:41:24.781215 [ 19 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request | ||
| 167 | 2025.05.14 01:41:24.781236 [ 19 ] {} <Trace> RaftInstance: sent | ||
| 168 | 2025.05.14 01:41:24.781246 [ 19 ] {} <Information> RaftInstance: === LEADER (term 2) === | ||
| 169 | 2025.05.14 01:41:24.781296 [ 10 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 170 | 2025.05.14 01:41:24.781317 [ 10 ] {} <Trace> RaftInstance: quorum idx 1, 1 0 0 | ||
| 171 | 2025.05.14 01:41:24.781326 [ 10 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 0, quick_commit_index_ 0, state_->get_commit_idx() 0 | ||
| 172 | 2025.05.14 01:41:24.781385 [ 14 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 3 with Result=1, Term=2, NextIndex=1 | ||
| 173 | 2025.05.14 01:41:24.781436 [ 14 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2 | ||
| 174 | 2025.05.14 01:41:24.781523 [ 14 ] {} <Information> RaftInstance: Election completed, will ignore the voting result from this server | ||
| 175 | 2025.05.14 01:41:24.782187 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK | ||
| 176 | 2025.05.14 01:41:24.782212 [ 12 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=2, NextIndex=2 | ||
| 177 | 2025.05.14 01:41:24.782231 [ 12 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2 | ||
| 178 | 2025.05.14 01:41:24.782240 [ 12 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 | ||
| 179 | 2025.05.14 01:41:24.782247 [ 12 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes | ||
| 180 | 2025.05.14 01:41:24.782256 [ 12 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 0, new matched idx: 1 | ||
| 181 | 2025.05.14 01:41:24.782266 [ 12 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 182 | 2025.05.14 01:41:24.782278 [ 12 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 | ||
| 183 | 2025.05.14 01:41:24.782285 [ 12 ] {} <Debug> RaftInstance: trigger commit upto 1 | ||
| 184 | 2025.05.14 01:41:24.782291 [ 12 ] {} <Trace> RaftInstance: send request to 3 | ||
| 185 | 2025.05.14 01:41:24.782297 [ 12 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 | ||
| 186 | 2025.05.14 01:41:24.782305 [ 12 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 1 | ||
| 187 | 2025.05.14 01:41:24.782322 [ 12 ] {} <Trace> RaftInstance: EMPTY PAYLOAD | ||
| 188 | 2025.05.14 01:41:24.782328 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request | ||
| 189 | 2025.05.14 01:41:24.782361 [ 12 ] {} <Trace> RaftInstance: sent | ||
| 190 | 2025.05.14 01:41:24.782371 [ 12 ] {} <Debug> RaftInstance: Server 2 is busy, skip the request | ||
| 191 | 2025.05.14 01:41:24.782378 [ 12 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0 | ||
| 192 | 2025.05.14 01:41:24.782392 [ 12 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread) | ||
| 193 | 2025.05.14 01:41:24.782458 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK | ||
| 194 | 2025.05.14 01:41:24.782458 [ 20 ] {} <Trace> RaftInstance: commit_cv_ wake up | ||
| 195 | 2025.05.14 01:41:24.782472 [ 12 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=2, NextIndex=2 | ||
| 196 | 2025.05.14 01:41:24.782556 [ 12 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2 | ||
| 197 | 2025.05.14 01:41:24.782570 [ 20 ] {} <Debug> RaftInstance: commit upto 1, current idx 0 | ||
| 198 | 2025.05.14 01:41:24.782574 [ 12 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 | ||
| 199 | 2025.05.14 01:41:24.782590 [ 20 ] {} <Trace> RaftInstance: commit upto 1, current idx 1 | ||
| 200 | 2025.05.14 01:41:24.782604 [ 12 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes | ||
| 201 | 2025.05.14 01:41:24.782623 [ 12 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 0, new matched idx: 1 | ||
| 202 | 2025.05.14 01:41:24.782630 [ 12 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 203 | 2025.05.14 01:41:24.782642 [ 12 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 204 | 2025.05.14 01:41:24.782650 [ 12 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0 | ||
| 205 | 2025.05.14 01:41:24.782658 [ 12 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread) | ||
| 206 | 2025.05.14 01:41:24.782666 [ 12 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 2 | ||
| 207 | 2025.05.14 01:41:24.782675 [ 12 ] {} <Trace> RaftInstance: send request to 2 | ||
| 208 | 2025.05.14 01:41:24.782682 [ 12 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 | ||
| 209 | 2025.05.14 01:41:24.782691 [ 12 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 1 | ||
| 210 | 2025.05.14 01:41:24.782698 [ 12 ] {} <Trace> RaftInstance: EMPTY PAYLOAD | ||
| 211 | 2025.05.14 01:41:24.782704 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request | ||
| 212 | 2025.05.14 01:41:24.782725 [ 12 ] {} <Trace> RaftInstance: sent | ||
| 213 | 2025.05.14 01:41:24.782759 [ 20 ] {} <Information> RaftInstance: config at index 1 is committed, prev config log idx 0 | ||
| 214 | 2025.05.14 01:41:24.782816 [ 20 ] {} <Information> RaftInstance: new config log idx 1, prev log idx 0, cur config log idx 0, prev log idx 0 | ||
| 215 | 2025.05.14 01:41:24.782855 [ 20 ] {} <Debug> RaftInstance: system is reconfigured to have 3 servers, last config index: 0, this config index: 1 | ||
| 216 | 2025.05.14 01:41:24.782893 [ 15 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK | ||
| 217 | 2025.05.14 01:41:24.782900 [ 20 ] {} <Information> RaftInstance: new configuration: log idx 1, prev log idx 0 | ||
| 218 | peer 1, DC ID 0, ::1:9234, voting member, 1 | ||
| 219 | peer 2, DC ID 0, ::1:9235, voting member, 1 | ||
| 220 | peer 3, DC ID 0, ::1:9236, voting member, 1 | ||
| 221 | my id: 1, leader: 1, term: 2 | ||
| 222 | 2025.05.14 01:41:24.782981 [ 20 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 223 | 2025.05.14 01:41:24.783019 [ 20 ] {} <Debug> RaftInstance: DONE: commit upto 1, current idx 1 | ||
| 224 | 2025.05.14 01:41:24.783055 [ 15 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=2, NextIndex=2 | ||
| 225 | 2025.05.14 01:41:24.783070 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep | ||
| 226 | 2025.05.14 01:41:24.783084 [ 15 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2 | ||
| 227 | 2025.05.14 01:41:24.783104 [ 15 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 | ||
| 228 | 2025.05.14 01:41:24.783115 [ 15 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes | ||
| 229 | 2025.05.14 01:41:24.783109 [ 1 ] {} <Debug> KeeperDispatcher: Quorum initialized | ||
| 230 | 2025.05.14 01:41:24.783134 [ 15 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1 | ||
| 231 | 2025.05.14 01:41:24.783173 [ 15 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 232 | 2025.05.14 01:41:24.783210 [ 15 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 233 | 2025.05.14 01:41:24.783219 [ 15 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 234 | 2025.05.14 01:41:24.783377 [ 1 ] {} <Debug> KeeperDispatcher: Dispatcher initialized | ||
| 235 | 2025.05.14 01:41:24.784142 [ 1 ] {} <Information> Application: Listening for Keeper (tcp): [::]:9181 | ||
| 236 | 2025.05.14 01:41:24.784239 [ 1 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 180.87 KiB, peak 2.17 MiB, free memory in arenas 0.00 B, will set to 118.55 MiB (RSS), difference: 118.38 MiB | ||
| 237 | 2025.05.14 01:41:24.785453 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml' | ||
| 238 | 2025.05.14 01:41:24.786087 [ 13 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK | ||
| 239 | 2025.05.14 01:41:24.786131 [ 13 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=2, NextIndex=2 | ||
| 240 | 2025.05.14 01:41:24.786144 [ 13 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2 | ||
| 241 | 2025.05.14 01:41:24.786155 [ 13 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 | ||
| 242 | 2025.05.14 01:41:24.786181 [ 13 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes | ||
| 243 | 2025.05.14 01:41:24.786189 [ 13 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1 | ||
| 244 | 2025.05.14 01:41:24.786200 [ 13 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 245 | 2025.05.14 01:41:24.786219 [ 13 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 246 | 2025.05.14 01:41:24.786227 [ 13 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 247 | 2025.05.14 01:41:24.786566 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml', performing update on configuration | ||
| 248 | 2025.05.14 01:41:24.786674 [ 1 ] {} <Trace> KeeperDispatcher: Configuration update triggered, but nothing changed for Raft | ||
| 249 | 2025.05.14 01:41:24.786726 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
| 250 | 2025.05.14 01:41:24.786749 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml', performed update on configuration | ||
| 251 | 2025.05.14 01:41:24.787543 [ 1 ] {} <Information> Application: Ready for connections. | ||
| 252 | 2025.05.14 01:41:25.000380 [ 26 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 120.78 MiB, peak 127.25 MiB, free memory in arenas 0.00 B, will set to 118.79 MiB (RSS), difference: -1.99 MiB | ||
| 253 | 2025.05.14 01:41:25.281262 [ 19 ] {} <Debug> RaftInstance: heartbeat timeout for 3 | ||
| 254 | 2025.05.14 01:41:25.281342 [ 19 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 255 | 2025.05.14 01:41:25.281360 [ 19 ] {} <Trace> RaftInstance: send request to 3 | ||
| 256 | 2025.05.14 01:41:25.281373 [ 19 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 | ||
| 257 | 2025.05.14 01:41:25.281387 [ 19 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 2 | ||
| 258 | 2025.05.14 01:41:25.281412 [ 19 ] {} <Trace> RaftInstance: EMPTY PAYLOAD | ||
| 259 | 2025.05.14 01:41:25.281432 [ 19 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request | ||
| 260 | 2025.05.14 01:41:25.281562 [ 19 ] {} <Trace> RaftInstance: sent | ||
| 261 | 2025.05.14 01:41:25.281675 [ 13 ] {} <Debug> RaftInstance: heartbeat timeout for 2 | ||
| 262 | 2025.05.14 01:41:25.281733 [ 13 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 263 | 2025.05.14 01:41:25.281748 [ 13 ] {} <Trace> RaftInstance: send request to 2 | ||
| 264 | 2025.05.14 01:41:25.281762 [ 13 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2 | ||
| 265 | 2025.05.14 01:41:25.281797 [ 13 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 2 | ||
| 266 | 2025.05.14 01:41:25.281807 [ 13 ] {} <Trace> RaftInstance: EMPTY PAYLOAD | ||
| 267 | 2025.05.14 01:41:25.281816 [ 13 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request | ||
| 268 | 2025.05.14 01:41:25.281881 [ 13 ] {} <Trace> RaftInstance: sent | ||
| 269 | 2025.05.14 01:41:25.281948 [ 15 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK | ||
| 270 | 2025.05.14 01:41:25.282033 [ 15 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=2, NextIndex=2 | ||
| 271 | 2025.05.14 01:41:25.282046 [ 15 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2 | ||
| 272 | 2025.05.14 01:41:25.282058 [ 15 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2 | ||
| 273 | 2025.05.14 01:41:25.282067 [ 15 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes | ||
| 274 | 2025.05.14 01:41:25.282130 [ 15 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1 | ||
| 275 | 2025.05.14 01:41:25.282149 [ 15 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 276 | 2025.05.14 01:41:25.282161 [ 15 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 277 | 2025.05.14 01:41:25.282169 [ 15 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 278 | 2025.05.14 01:41:25.282353 [ 13 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK | ||
| 279 | 2025.05.14 01:41:25.282405 [ 13 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=2, NextIndex=2 | ||
| 280 | 2025.05.14 01:41:25.282416 [ 13 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2 | ||
| 281 | 2025.05.14 01:41:25.282429 [ 13 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2 | ||
| 282 | 2025.05.14 01:41:25.282439 [ 13 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes | ||
| 283 | 2025.05.14 01:41:25.282455 [ 13 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1 | ||
| 284 | 2025.05.14 01:41:25.282467 [ 13 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1 | ||
| 285 | 2025.05.14 01:41:25.282478 [ 13 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 | ||
| 286 | 2025.05.14 01:41:25.282486 [ 13 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||