2025.05.14 01:41:21.494130 [ 1 ] {} <Information> : Starting ClickHouse Keeper 23.8.7.1(revision : 54477, git hash: 3042d295d963012962f5c683bd2776fa331a38c3, build id: <unknown>), PID 16476
2025.05.14 01:41:21.494370 [ 1 ] {} <Information> Application: starting up
2025.05.14 01:41:21.494424 [ 1 ] {} <Information> Application: OS Name = SunOS, OS Version = 5.11, OS Architecture = i86pc
2025.05.14 01:41:21.495977 [ 1 ] {} <Debug> Application: Initializing DateLUT.
2025.05.14 01:41:21.496004 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
2025.05.14 01:41:21.499704 [ 1 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously
2025.05.14 01:41:21.499797 [ 1 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher
2025.05.14 01:41:21.500302 [ 1 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled
2025.05.14 01:41:21.500321 [ 1 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled
2025.05.14 01:41:21.500343 [ 1 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled
2025.05.14 01:41:21.500783 [ 1 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk
2025.05.14 01:41:21.501281 [ 1 ] {} <Trace> KeeperLogStore: Reading from disk LocalLogDisk
2025.05.14 01:41:21.501353 [ 1 ] {} <Warning> KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.1-clickhouse-iFcRob/log. It's Ok if it's the first run of clickhouse-keeper.
2025.05.14 01:41:21.501563 [ 1 ] {} <Information> KeeperLogStore: force_sync enabled
2025.05.14 01:41:21.501579 [ 1 ] {} <Debug> KeeperDispatcher: Waiting server to initialize
2025.05.14 01:41:21.501588 [ 1 ] {} <Debug> KeeperStateMachine: Totally have 0 snapshots
2025.05.14 01:41:21.501600 [ 1 ] {} <Debug> KeeperStateMachine: No existing snapshots, last committed log index 0
2025.05.14 01:41:21.501667 [ 1 ] {} <Warning> KeeperLogStore: Removing all changelogs
2025.05.14 01:41:21.501759 [ 1 ] {} <Trace> Changelog: Starting new changelog changelog_1_100000.bin.zstd
2025.05.14 01:41:21.501940 [ 1 ] {} <Information> KeeperServer: Preprocessing 0 log entries
2025.05.14 01:41:21.501956 [ 1 ] {} <Information> KeeperServer: Preprocessing done
2025.05.14 01:41:21.501986 [ 1 ] {} <Information> KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk
2025.05.14 01:41:21.504664 [ 1 ] {} <Information> RaftInstance: Raft ASIO listener initiated on :::9234, unsecured
2025.05.14 01:41:21.504942 [ 1 ] {} <Warning> KeeperStateManager: No state was read
2025.05.14 01:41:21.505183 [ 1 ] {} <Information> RaftInstance: parameters: election timeout range 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 100000, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on
2025.05.14 01:41:21.505246 [ 1 ] {} <Information> RaftInstance: new election timeout range: 1000 - 2000
2025.05.14 01:41:21.505312 [ 1 ] {} <Information> RaftInstance:    === INIT RAFT SERVER ===
commit index 0
term 0
election timer allowed
log store start 1, end 0
config log idx 0, prev log idx 0
2025.05.14 01:41:21.505497 [ 1 ] {} <Trace> RaftInstance: asio client created: 1bc071e0
2025.05.14 01:41:21.505543 [ 1 ] {} <Trace> RaftInstance: asio client created: 1bc06060
2025.05.14 01:41:21.505560 [ 1 ] {} <Information> RaftInstance: peer 1: DC ID 0, ::1:9234, voting member, 1
peer 2: DC ID 0, ::1:9235, voting member, 1
peer 3: DC ID 0, ::1:9236, voting member, 1
my id: 1, voting_member
num peers: 2
2025.05.14 01:41:21.505576 [ 1 ] {} <Information> RaftInstance: global manager does not exist. will use local thread for commit and append
2025.05.14 01:41:21.505704 [ 1 ] {} <Information> RaftInstance: wait for HB, for 50 + [1000, 2000] ms
2025.05.14 01:41:21.505779 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep
2025.05.14 01:41:21.505830 [ 21 ] {} <Information> RaftInstance: bg append_entries thread initiated
2025.05.14 01:41:21.556281 [ 1 ] {} <Trace> RaftInstance: re-schedule election timer
2025.05.14 01:41:21.556340 [ 1 ] {} <Debug> RaftInstance: server 1 started
2025.05.14 01:41:21.556414 [ 1 ] {} <Trace> RaftInstance: asio rpc session created: 1c9199a0
2025.05.14 01:41:21.556510 [ 1 ] {} <Debug> KeeperDispatcher: Server initialized, waiting for quorum
2025.05.14 01:41:23.378180 [ 12 ] {} <Trace> RaftInstance: election timeout
2025.05.14 01:41:23.378236 [ 12 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2025.05.14 01:41:23.378286 [ 12 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
2025.05.14 01:41:23.378303 [ 12 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
2025.05.14 01:41:23.378361 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 1, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
2025.05.14 01:41:23.378381 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type pre_vote_request
2025.05.14 01:41:23.378392 [ 12 ] {} <Debug> RaftInstance: socket 1bc06060 to ::1:9236 is not opened yet
2025.05.14 01:41:23.378575 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 2, type pre_vote_request
2025.05.14 01:41:23.378596 [ 12 ] {} <Debug> RaftInstance: socket 1bc071e0 to ::1:9235 is not opened yet
2025.05.14 01:41:23.378606 [ 12 ] {} <Trace> RaftInstance: cancel existing timer
2025.05.14 01:41:23.378612 [ 12 ] {} <Trace> RaftInstance: re-schedule election timer
2025.05.14 01:41:23.379650 [ 19 ] {} <Information> RaftInstance: 1bc06060 connected to ::1:9236 (as a client)
2025.05.14 01:41:23.379706 [ 12 ] {} <Information> RaftInstance: receive a incoming rpc connection
2025.05.14 01:41:23.379769 [ 12 ] {} <Information> RaftInstance: session 1 got connection from ::1:41690 (as a server)
2025.05.14 01:41:23.379788 [ 19 ] {} <Information> RaftInstance: 1bc071e0 connected to ::1:9235 (as a client)
2025.05.14 01:41:23.379879 [ 12 ] {} <Trace> RaftInstance: asio rpc session created: 1c9190a0
2025.05.14 01:41:23.380250 [ 17 ] {} <Debug> RaftInstance: Receive a pre_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0
2025.05.14 01:41:23.380312 [ 17 ] {} <Information> RaftInstance: [PRE-VOTE REQ] my role candidate, from peer 2, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 0 / mine 0
HB dead
2025.05.14 01:41:23.380325 [ 17 ] {} <Information> RaftInstance: pre-vote decision: O (grant)
2025.05.14 01:41:23.380337 [ 17 ] {} <Debug> RaftInstance: Response back a pre_vote_response message to 2 with Accepted=1, Term=0, NextIndex=1
2025.05.14 01:41:23.380553 [ 16 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type pre_vote_request, OK
2025.05.14 01:41:23.380643 [ 16 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 2 with Result=1, Term=0, NextIndex=1
2025.05.14 01:41:23.380656 [ 16 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 0
2025.05.14 01:41:23.380670 [ 16 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 0, resp term 0, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2025.05.14 01:41:23.380682 [ 16 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0
2025.05.14 01:41:23.380694 [ 16 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote
2025.05.14 01:41:23.380971 [ 16 ] {} <Information> RaftInstance: [VOTE INIT] my id 1, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1)
2025.05.14 01:41:23.380994 [ 16 ] {} <Debug> RaftInstance: send request_vote_request to server 3 with term 1
2025.05.14 01:41:23.380993 [ 17 ] {} <Debug> RaftInstance: Receive a request_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=1
2025.05.14 01:41:23.381012 [ 16 ] {} <Warning> RaftInstance: failed to send vote request: peer 3 (::1:9236) is busy
2025.05.14 01:41:23.381051 [ 16 ] {} <Debug> RaftInstance: send request_vote_request to server 2 with term 1
2025.05.14 01:41:23.381057 [ 16 ] {} <Trace> RaftInstance: send req 1 -> 2, type request_vote_request
2025.05.14 01:41:23.381123 [ 16 ] {} <Trace> RaftInstance: cancel existing timer
2025.05.14 01:41:23.381151 [ 16 ] {} <Trace> RaftInstance: re-schedule election timer
2025.05.14 01:41:23.381291 [ 17 ] {} <Information> RaftInstance: [VOTE REQ] my role candidate, from peer 2, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 1 / mine 1
priority: target 1 / mine 1, voted_for 1
2025.05.14 01:41:23.381301 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type request_vote_request, OK
2025.05.14 01:41:23.381321 [ 17 ] {} <Information> RaftInstance: decision: X (deny), term 1
2025.05.14 01:41:23.381357 [ 17 ] {} <Debug> RaftInstance: Response back a request_vote_response message to 2 with Accepted=0, Term=1, NextIndex=0
2025.05.14 01:41:23.381367 [ 12 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 2 with Result=0, Term=1, NextIndex=0
2025.05.14 01:41:23.381366 [ 14 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type pre_vote_request, OK
2025.05.14 01:41:23.381381 [ 12 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1
2025.05.14 01:41:23.381465 [ 12 ] {} <Information> RaftInstance: [VOTE RESP] peer 2 (X), resp term 1, my role candidate, granted 1, responded 2, num voting members 3, quorum 2
2025.05.14 01:41:23.381525 [ 14 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 3 with Result=1, Term=0, NextIndex=1
2025.05.14 01:41:23.381551 [ 14 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 0
2025.05.14 01:41:23.381564 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 3 (O), term 0, resp term 0, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
2025.05.14 01:41:23.381587 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0
2025.05.14 01:41:23.381612 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing
2025.05.14 01:41:24.333356 [ 17 ] {} <Information> RaftInstance: receive a incoming rpc connection
2025.05.14 01:41:24.333420 [ 17 ] {} <Information> RaftInstance: session 2 got connection from ::1:64933 (as a server)
2025.05.14 01:41:24.333533 [ 17 ] {} <Trace> RaftInstance: asio rpc session created: 1c94f6e0
2025.05.14 01:41:24.333573 [ 14 ] {} <Debug> RaftInstance: Receive a pre_vote_request message from 3 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0
2025.05.14 01:41:24.333637 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE REQ] my role candidate, from peer 3, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 0 / mine 1
HB dead
2025.05.14 01:41:24.333646 [ 14 ] {} <Information> RaftInstance: pre-vote decision: O (grant)
2025.05.14 01:41:24.333669 [ 14 ] {} <Debug> RaftInstance: Response back a pre_vote_response message to 3 with Accepted=1, Term=0, NextIndex=1
2025.05.14 01:41:24.334350 [ 17 ] {} <Debug> RaftInstance: Receive a request_vote_request message from 3 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=1
2025.05.14 01:41:24.334392 [ 17 ] {} <Information> RaftInstance: [VOTE REQ] my role candidate, from peer 3, log term: req 0 / mine 0
last idx: req 0 / mine 0, term: req 1 / mine 1
priority: target 1 / mine 1, voted_for 1
2025.05.14 01:41:24.334415 [ 17 ] {} <Information> RaftInstance: decision: X (deny), term 1
2025.05.14 01:41:24.334424 [ 17 ] {} <Debug> RaftInstance: Response back a request_vote_response message to 3 with Accepted=0, Term=1, NextIndex=0
2025.05.14 01:41:24.777912 [ 14 ] {} <Trace> RaftInstance: election timeout
2025.05.14 01:41:24.777991 [ 14 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
2025.05.14 01:41:24.778050 [ 14 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
2025.05.14 01:41:24.778067 [ 14 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 1, target p 1, my p 1, hb dead, pre-vote done
2025.05.14 01:41:24.778079 [ 14 ] {} <Information> RaftInstance: pre-vote term (0) is different, reset it to 1
2025.05.14 01:41:24.778100 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 1, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1)
2025.05.14 01:41:24.778121 [ 14 ] {} <Trace> RaftInstance: send req 1 -> 3, type pre_vote_request
2025.05.14 01:41:24.778181 [ 14 ] {} <Trace> RaftInstance: send req 1 -> 2, type pre_vote_request
2025.05.14 01:41:24.778266 [ 14 ] {} <Trace> RaftInstance: cancel existing timer
2025.05.14 01:41:24.778280 [ 14 ] {} <Trace> RaftInstance: re-schedule election timer
2025.05.14 01:41:24.778459 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type pre_vote_request, OK
2025.05.14 01:41:24.778526 [ 12 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 3 with Result=1, Term=1, NextIndex=1
2025.05.14 01:41:24.778534 [ 12 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 1
2025.05.14 01:41:24.778546 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 3 (O), term 1, resp term 1, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
2025.05.14 01:41:24.778557 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 1
2025.05.14 01:41:24.778566 [ 14 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type pre_vote_request, OK
2025.05.14 01:41:24.778569 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote
2025.05.14 01:41:24.778751 [ 12 ] {} <Debug> IDisk: Copying from LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.1-clickhouse-iFcRob/snapshots) state to LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.1-clickhouse-iFcRob/snapshots) state-OLD.
2025.05.14 01:41:24.779153 [ 12 ] {} <Information> RaftInstance: [VOTE INIT] my id 1, my role candidate, term 2, log idx 0, log term 0, priority (target 1 / mine 1)
2025.05.14 01:41:24.779176 [ 12 ] {} <Debug> RaftInstance: send request_vote_request to server 3 with term 2
2025.05.14 01:41:24.779184 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type request_vote_request
2025.05.14 01:41:24.779250 [ 12 ] {} <Debug> RaftInstance: send request_vote_request to server 2 with term 2
2025.05.14 01:41:24.779258 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 2, type request_vote_request
2025.05.14 01:41:24.779299 [ 12 ] {} <Trace> RaftInstance: cancel existing timer
2025.05.14 01:41:24.779322 [ 12 ] {} <Trace> RaftInstance: re-schedule election timer
2025.05.14 01:41:24.779377 [ 14 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 2 with Result=1, Term=1, NextIndex=1
2025.05.14 01:41:24.779423 [ 14 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 1
2025.05.14 01:41:24.779439 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 1, resp term 1, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
2025.05.14 01:41:24.779452 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 1
2025.05.14 01:41:24.779463 [ 14 ] {} <Information> RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing
2025.05.14 01:41:24.780407 [ 19 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type request_vote_request, OK
2025.05.14 01:41:24.780443 [ 19 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 2 with Result=1, Term=2, NextIndex=1
2025.05.14 01:41:24.780454 [ 19 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2
2025.05.14 01:41:24.780466 [ 19 ] {} <Information> RaftInstance: [VOTE RESP] peer 2 (O), resp term 2, my role candidate, granted 2, responded 2, num voting members 3, quorum 2
2025.05.14 01:41:24.780484 [ 19 ] {} <Information> RaftInstance: Server is elected as leader for term 2
2025.05.14 01:41:24.780518 [ 19 ] {} <Information> RaftInstance: number of pending commit elements: 0
2025.05.14 01:41:24.780543 [ 19 ] {} <Information> RaftInstance: state machine commit index 0, precommit index 0, last log index 0
2025.05.14 01:41:24.780555 [ 19 ] {} <Trace> RaftInstance: peer 3, interval: 500
2025.05.14 01:41:24.780567 [ 19 ] {} <Trace> RaftInstance: peer 2, interval: 500
2025.05.14 01:41:24.780574 [ 14 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type request_vote_request, OK
2025.05.14 01:41:24.780589 [ 19 ] {} <Information> RaftInstance: [BECOME LEADER] appended new config at 1
2025.05.14 01:41:24.781027 [ 19 ] {} <Trace> RaftInstance: send request to 3
2025.05.14 01:41:24.781042 [ 10 ] {} <Trace> RaftInstance: got log append completion notification: OK
2025.05.14 01:41:24.781052 [ 19 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2025.05.14 01:41:24.781081 [ 19 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=2, peer_last_sent_idx 0
2025.05.14 01:41:24.781088 [ 19 ] {} <Debug> RaftInstance: idx: 1
2025.05.14 01:41:24.781100 [ 19 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request
2025.05.14 01:41:24.781174 [ 19 ] {} <Trace> RaftInstance: sent
2025.05.14 01:41:24.781183 [ 19 ] {} <Trace> RaftInstance: send request to 2
2025.05.14 01:41:24.781189 [ 19 ] {} <Trace> RaftInstance: last_log_idx: 0, starting_idx: 1, cur_nxt_idx: 2
2025.05.14 01:41:24.781202 [ 19 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=0, LastLogTerm=0, EntriesLength=1, CommitIndex=0, Term=2, peer_last_sent_idx 0
2025.05.14 01:41:24.781208 [ 19 ] {} <Debug> RaftInstance: idx: 1
2025.05.14 01:41:24.781215 [ 19 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request
2025.05.14 01:41:24.781236 [ 19 ] {} <Trace> RaftInstance: sent
2025.05.14 01:41:24.781246 [ 19 ] {} <Information> RaftInstance:   === LEADER (term 2) ===
2025.05.14 01:41:24.781296 [ 10 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1
2025.05.14 01:41:24.781317 [ 10 ] {} <Trace> RaftInstance: quorum idx 1, 1 0 0 
2025.05.14 01:41:24.781326 [ 10 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 0, quick_commit_index_ 0, state_->get_commit_idx() 0
2025.05.14 01:41:24.781385 [ 14 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 3 with Result=1, Term=2, NextIndex=1
2025.05.14 01:41:24.781436 [ 14 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2
2025.05.14 01:41:24.781523 [ 14 ] {} <Information> RaftInstance: Election completed, will ignore the voting result from this server
2025.05.14 01:41:24.782187 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK
2025.05.14 01:41:24.782212 [ 12 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=2, NextIndex=2
2025.05.14 01:41:24.782231 [ 12 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2
2025.05.14 01:41:24.782240 [ 12 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2025.05.14 01:41:24.782247 [ 12 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2025.05.14 01:41:24.782256 [ 12 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 0, new matched idx: 1
2025.05.14 01:41:24.782266 [ 12 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1
2025.05.14 01:41:24.782278 [ 12 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 0 
2025.05.14 01:41:24.782285 [ 12 ] {} <Debug> RaftInstance: trigger commit upto 1
2025.05.14 01:41:24.782291 [ 12 ] {} <Trace> RaftInstance: send request to 3
2025.05.14 01:41:24.782297 [ 12 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2025.05.14 01:41:24.782305 [ 12 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 1
2025.05.14 01:41:24.782322 [ 12 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2025.05.14 01:41:24.782328 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request
2025.05.14 01:41:24.782361 [ 12 ] {} <Trace> RaftInstance: sent
2025.05.14 01:41:24.782371 [ 12 ] {} <Debug> RaftInstance: Server 2 is busy, skip the request
2025.05.14 01:41:24.782378 [ 12 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0
2025.05.14 01:41:24.782392 [ 12 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2025.05.14 01:41:24.782458 [ 12 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK
2025.05.14 01:41:24.782458 [ 20 ] {} <Trace> RaftInstance: commit_cv_ wake up
2025.05.14 01:41:24.782472 [ 12 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=2, NextIndex=2
2025.05.14 01:41:24.782556 [ 12 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2
2025.05.14 01:41:24.782570 [ 20 ] {} <Debug> RaftInstance: commit upto 1, current idx 0
2025.05.14 01:41:24.782574 [ 12 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2
2025.05.14 01:41:24.782590 [ 20 ] {} <Trace> RaftInstance: commit upto 1, current idx 1
2025.05.14 01:41:24.782604 [ 12 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes
2025.05.14 01:41:24.782623 [ 12 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 0, new matched idx: 1
2025.05.14 01:41:24.782630 [ 12 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1
2025.05.14 01:41:24.782642 [ 12 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2025.05.14 01:41:24.782650 [ 12 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0
2025.05.14 01:41:24.782658 [ 12 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread)
2025.05.14 01:41:24.782666 [ 12 ] {} <Debug> RaftInstance: reqeust append entries need to catchup, p 2
2025.05.14 01:41:24.782675 [ 12 ] {} <Trace> RaftInstance: send request to 2
2025.05.14 01:41:24.782682 [ 12 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2025.05.14 01:41:24.782691 [ 12 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 1
2025.05.14 01:41:24.782698 [ 12 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2025.05.14 01:41:24.782704 [ 12 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request
2025.05.14 01:41:24.782725 [ 12 ] {} <Trace> RaftInstance: sent
2025.05.14 01:41:24.782759 [ 20 ] {} <Information> RaftInstance: config at index 1 is committed, prev config log idx 0
2025.05.14 01:41:24.782816 [ 20 ] {} <Information> RaftInstance: new config log idx 1, prev log idx 0, cur config log idx 0, prev log idx 0
2025.05.14 01:41:24.782855 [ 20 ] {} <Debug> RaftInstance: system is reconfigured to have 3 servers, last config index: 0, this config index: 1
2025.05.14 01:41:24.782893 [ 15 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK
2025.05.14 01:41:24.782900 [ 20 ] {} <Information> RaftInstance: new configuration: log idx 1, prev log idx 0
peer 1, DC ID 0, ::1:9234, voting member, 1
peer 2, DC ID 0, ::1:9235, voting member, 1
peer 3, DC ID 0, ::1:9236, voting member, 1
my id: 1, leader: 1, term: 2
2025.05.14 01:41:24.782981 [ 20 ] {} <Trace> RaftInstance: (update) new target priority: 1
2025.05.14 01:41:24.783019 [ 20 ] {} <Debug> RaftInstance: DONE: commit upto 1, current idx 1
2025.05.14 01:41:24.783055 [ 15 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=2, NextIndex=2
2025.05.14 01:41:24.783070 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep
2025.05.14 01:41:24.783084 [ 15 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2
2025.05.14 01:41:24.783104 [ 15 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2025.05.14 01:41:24.783115 [ 15 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2025.05.14 01:41:24.783109 [ 1 ] {} <Debug> KeeperDispatcher: Quorum initialized
2025.05.14 01:41:24.783134 [ 15 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2025.05.14 01:41:24.783173 [ 15 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1
2025.05.14 01:41:24.783210 [ 15 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2025.05.14 01:41:24.783219 [ 15 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2025.05.14 01:41:24.783377 [ 1 ] {} <Debug> KeeperDispatcher: Dispatcher initialized
2025.05.14 01:41:24.784142 [ 1 ] {} <Information> Application: Listening for Keeper (tcp): [::]:9181
2025.05.14 01:41:24.784239 [ 1 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 180.87 KiB, peak 2.17 MiB, free memory in arenas 0.00 B, will set to 118.55 MiB (RSS), difference: 118.38 MiB
2025.05.14 01:41:24.785453 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml'
2025.05.14 01:41:24.786087 [ 13 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK
2025.05.14 01:41:24.786131 [ 13 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=2, NextIndex=2
2025.05.14 01:41:24.786144 [ 13 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2
2025.05.14 01:41:24.786155 [ 13 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2
2025.05.14 01:41:24.786181 [ 13 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes
2025.05.14 01:41:24.786189 [ 13 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1
2025.05.14 01:41:24.786200 [ 13 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1
2025.05.14 01:41:24.786219 [ 13 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2025.05.14 01:41:24.786227 [ 13 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2025.05.14 01:41:24.786566 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml', performing update on configuration
2025.05.14 01:41:24.786674 [ 1 ] {} <Trace> KeeperDispatcher: Configuration update triggered, but nothing changed for Raft
2025.05.14 01:41:24.786726 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
2025.05.14 01:41:24.786749 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml', performed update on configuration
2025.05.14 01:41:24.787543 [ 1 ] {} <Information> Application: Ready for connections.
2025.05.14 01:41:25.000380 [ 26 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 120.78 MiB, peak 127.25 MiB, free memory in arenas 0.00 B, will set to 118.79 MiB (RSS), difference: -1.99 MiB
2025.05.14 01:41:25.281262 [ 19 ] {} <Debug> RaftInstance: heartbeat timeout for 3
2025.05.14 01:41:25.281342 [ 19 ] {} <Trace> RaftInstance: (update) new target priority: 1
2025.05.14 01:41:25.281360 [ 19 ] {} <Trace> RaftInstance: send request to 3
2025.05.14 01:41:25.281373 [ 19 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2025.05.14 01:41:25.281387 [ 19 ] {} <Debug> RaftInstance: append_entries for 3 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 2
2025.05.14 01:41:25.281412 [ 19 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2025.05.14 01:41:25.281432 [ 19 ] {} <Trace> RaftInstance: send req 1 -> 3, type append_entries_request
2025.05.14 01:41:25.281562 [ 19 ] {} <Trace> RaftInstance: sent
2025.05.14 01:41:25.281675 [ 13 ] {} <Debug> RaftInstance: heartbeat timeout for 2
2025.05.14 01:41:25.281733 [ 13 ] {} <Trace> RaftInstance: (update) new target priority: 1
2025.05.14 01:41:25.281748 [ 13 ] {} <Trace> RaftInstance: send request to 2
2025.05.14 01:41:25.281762 [ 13 ] {} <Trace> RaftInstance: last_log_idx: 1, starting_idx: 1, cur_nxt_idx: 2
2025.05.14 01:41:25.281797 [ 13 ] {} <Debug> RaftInstance: append_entries for 2 with LastLogIndex=1, LastLogTerm=2, EntriesLength=0, CommitIndex=1, Term=2, peer_last_sent_idx 2
2025.05.14 01:41:25.281807 [ 13 ] {} <Trace> RaftInstance: EMPTY PAYLOAD
2025.05.14 01:41:25.281816 [ 13 ] {} <Trace> RaftInstance: send req 1 -> 2, type append_entries_request
2025.05.14 01:41:25.281881 [ 13 ] {} <Trace> RaftInstance: sent
2025.05.14 01:41:25.281948 [ 15 ] {} <Trace> RaftInstance: resp of req 1 -> 3, type append_entries_request, OK
2025.05.14 01:41:25.282033 [ 15 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 3 with Result=1, Term=2, NextIndex=2
2025.05.14 01:41:25.282046 [ 15 ] {} <Trace> RaftInstance: src: 3, dst: 1, resp->get_term(): 2
2025.05.14 01:41:25.282058 [ 15 ] {} <Trace> RaftInstance: handle append entries resp (from 3), resp.get_next_idx(): 2
2025.05.14 01:41:25.282067 [ 15 ] {} <Trace> RaftInstance: peer 3 batch size hint: 0 bytes
2025.05.14 01:41:25.282130 [ 15 ] {} <Trace> RaftInstance: peer 3, prev matched idx: 1, new matched idx: 1
2025.05.14 01:41:25.282149 [ 15 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1
2025.05.14 01:41:25.282161 [ 15 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2025.05.14 01:41:25.282169 [ 15 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1
2025.05.14 01:41:25.282353 [ 13 ] {} <Trace> RaftInstance: resp of req 1 -> 2, type append_entries_request, OK
2025.05.14 01:41:25.282405 [ 13 ] {} <Debug> RaftInstance: Receive a append_entries_response message from peer 2 with Result=1, Term=2, NextIndex=2
2025.05.14 01:41:25.282416 [ 13 ] {} <Trace> RaftInstance: src: 2, dst: 1, resp->get_term(): 2
2025.05.14 01:41:25.282429 [ 13 ] {} <Trace> RaftInstance: handle append entries resp (from 2), resp.get_next_idx(): 2
2025.05.14 01:41:25.282439 [ 13 ] {} <Trace> RaftInstance: peer 2 batch size hint: 0 bytes
2025.05.14 01:41:25.282455 [ 13 ] {} <Trace> RaftInstance: peer 2, prev matched idx: 1, new matched idx: 1
2025.05.14 01:41:25.282467 [ 13 ] {} <Trace> RaftInstance: last durable index 1, precommit index 1
2025.05.14 01:41:25.282478 [ 13 ] {} <Trace> RaftInstance: quorum idx 1, 1 1 1 
2025.05.14 01:41:25.282486 [ 13 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1