| 1 | 2025.05.14 01:41:22.832050 [ 1 ] {} <Information> : Starting ClickHouse Keeper 23.8.7.1(revision : 54477, git hash: 3042d295d963012962f5c683bd2776fa331a38c3, build id: <unknown>), PID 16482 | ||
| 2 | 2025.05.14 01:41:22.832281 [ 1 ] {} <Information> Application: starting up | ||
| 3 | 2025.05.14 01:41:22.832403 [ 1 ] {} <Information> Application: OS Name = SunOS, OS Version = 5.11, OS Architecture = i86pc | ||
| 4 | 2025.05.14 01:41:22.833692 [ 1 ] {} <Debug> Application: Initializing DateLUT. | ||
| 5 | 2025.05.14 01:41:22.833715 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'. | ||
| 6 | 2025.05.14 01:41:22.840767 [ 1 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously | ||
| 7 | 2025.05.14 01:41:22.841160 [ 1 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher | ||
| 8 | 2025.05.14 01:41:22.842528 [ 1 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled | ||
| 9 | 2025.05.14 01:41:22.842572 [ 1 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled | ||
| 10 | 2025.05.14 01:41:22.842596 [ 1 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled | ||
| 11 | 2025.05.14 01:41:22.842902 [ 1 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk | ||
| 12 | 2025.05.14 01:41:22.843362 [ 1 ] {} <Trace> KeeperLogStore: Reading from disk LocalLogDisk | ||
| 13 | 2025.05.14 01:41:22.843454 [ 1 ] {} <Warning> KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.3-clickhouse-y2UGSY/log. It's Ok if it's the first run of clickhouse-keeper. | ||
| 14 | 2025.05.14 01:41:22.843814 [ 1 ] {} <Information> KeeperLogStore: force_sync enabled | ||
| 15 | 2025.05.14 01:41:22.843858 [ 1 ] {} <Debug> KeeperDispatcher: Waiting server to initialize | ||
| 16 | 2025.05.14 01:41:22.843870 [ 1 ] {} <Debug> KeeperStateMachine: Totally have 0 snapshots | ||
| 17 | 2025.05.14 01:41:22.843880 [ 1 ] {} <Debug> KeeperStateMachine: No existing snapshots, last committed log index 0 | ||
| 18 | 2025.05.14 01:41:22.844094 [ 1 ] {} <Warning> KeeperLogStore: Removing all changelogs | ||
| 19 | 2025.05.14 01:41:22.844438 [ 1 ] {} <Trace> Changelog: Starting new changelog changelog_1_100000.bin.zstd | ||
| 20 | 2025.05.14 01:41:22.844677 [ 1 ] {} <Information> KeeperServer: Preprocessing 0 log entries | ||
| 21 | 2025.05.14 01:41:22.844699 [ 1 ] {} <Information> KeeperServer: Preprocessing done | ||
| 22 | 2025.05.14 01:41:22.844736 [ 1 ] {} <Information> KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk | ||
| 23 | 2025.05.14 01:41:22.847225 [ 1 ] {} <Information> RaftInstance: Raft ASIO listener initiated on :::9236, unsecured | ||
| 24 | 2025.05.14 01:41:22.847558 [ 1 ] {} <Warning> KeeperStateManager: No state was read | ||
| 25 | 2025.05.14 01:41:22.847636 [ 1 ] {} <Information> RaftInstance: parameters: election timeout range 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 100000, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on | ||
| 26 | 2025.05.14 01:41:22.847672 [ 1 ] {} <Information> RaftInstance: new election timeout range: 1000 - 2000 | ||
| 27 | 2025.05.14 01:41:22.847795 [ 1 ] {} <Information> RaftInstance: === INIT RAFT SERVER === | ||
| 28 | commit index 0 | ||
| 29 | term 0 | ||
| 30 | election timer allowed | ||
| 31 | log store start 1, end 0 | ||
| 32 | config log idx 0, prev log idx 0 | ||
| 33 | 2025.05.14 01:41:22.848126 [ 1 ] {} <Trace> RaftInstance: asio client created: 1bc071e0 | ||
| 34 | 2025.05.14 01:41:22.848196 [ 1 ] {} <Trace> RaftInstance: asio client created: 1bc06060 | ||
| 35 | 2025.05.14 01:41:22.848213 [ 1 ] {} <Information> RaftInstance: peer 1: DC ID 0, ::1:9234, voting member, 1 | ||
| 36 | peer 2: DC ID 0, ::1:9235, voting member, 1 | ||
| 37 | peer 3: DC ID 0, ::1:9236, voting member, 1 | ||
| 38 | my id: 3, voting_member | ||
| 39 | num peers: 2 | ||
| 40 | 2025.05.14 01:41:22.848235 [ 1 ] {} <Information> RaftInstance: global manager does not exist. will use local thread for commit and append | ||
| 41 | 2025.05.14 01:41:22.848399 [ 1 ] {} <Information> RaftInstance: wait for HB, for 50 + [1000, 2000] ms | ||
| 42 | 2025.05.14 01:41:22.848907 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep | ||
| 43 | 2025.05.14 01:41:22.849102 [ 21 ] {} <Information> RaftInstance: bg append_entries thread initiated | ||
| 44 | 2025.05.14 01:41:22.898797 [ 1 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 45 | 2025.05.14 01:41:22.898873 [ 1 ] {} <Debug> RaftInstance: server 3 started | ||
| 46 | 2025.05.14 01:41:22.898951 [ 1 ] {} <Trace> RaftInstance: asio rpc session created: 1c90d9a0 | ||
| 47 | 2025.05.14 01:41:22.899134 [ 1 ] {} <Debug> KeeperDispatcher: Server initialized, waiting for quorum | ||
| 48 | 2025.05.14 01:41:23.379603 [ 12 ] {} <Information> RaftInstance: receive a incoming rpc connection | ||
| 49 | 2025.05.14 01:41:23.379973 [ 12 ] {} <Information> RaftInstance: session 1 got connection from ::1:51312 (as a server) | ||
| 50 | 2025.05.14 01:41:23.380163 [ 12 ] {} <Trace> RaftInstance: asio rpc session created: 1c90d0a0 | ||
| 51 | 2025.05.14 01:41:23.380209 [ 12 ] {} <Information> RaftInstance: receive a incoming rpc connection | ||
| 52 | 2025.05.14 01:41:23.380227 [ 12 ] {} <Information> RaftInstance: session 2 got connection from ::1:36061 (as a server) | ||
| 53 | 2025.05.14 01:41:23.380264 [ 12 ] {} <Trace> RaftInstance: asio rpc session created: 1c94d6e0 | ||
| 54 | 2025.05.14 01:41:23.380331 [ 18 ] {} <Debug> RaftInstance: Receive a pre_vote_request message from 1 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0 | ||
| 55 | 2025.05.14 01:41:23.380619 [ 19 ] {} <Debug> RaftInstance: Receive a pre_vote_request message from 2 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=0 | ||
| 56 | 2025.05.14 01:41:23.381096 [ 18 ] {} <Information> RaftInstance: [PRE-VOTE REQ] my role follower, from peer 1, log term: req 0 / mine 0 | ||
| 57 | last idx: req 0 / mine 0, term: req 0 / mine 0 | ||
| 58 | HB dead | ||
| 59 | 2025.05.14 01:41:23.381119 [ 18 ] {} <Information> RaftInstance: pre-vote decision: O (grant) | ||
| 60 | 2025.05.14 01:41:23.381128 [ 18 ] {} <Debug> RaftInstance: Response back a pre_vote_response message to 1 with Accepted=1, Term=0, NextIndex=1 | ||
| 61 | 2025.05.14 01:41:23.381286 [ 19 ] {} <Information> RaftInstance: [PRE-VOTE REQ] my role follower, from peer 2, log term: req 0 / mine 0 | ||
| 62 | last idx: req 0 / mine 0, term: req 0 / mine 0 | ||
| 63 | HB dead | ||
| 64 | 2025.05.14 01:41:23.381313 [ 19 ] {} <Information> RaftInstance: pre-vote decision: O (grant) | ||
| 65 | 2025.05.14 01:41:23.381323 [ 19 ] {} <Debug> RaftInstance: Response back a pre_vote_response message to 2 with Accepted=1, Term=0, NextIndex=1 | ||
| 66 | 2025.05.14 01:41:24.331811 [ 12 ] {} <Trace> RaftInstance: election timeout | ||
| 67 | 2025.05.14 01:41:24.331879 [ 12 ] {} <Warning> RaftInstance: Election timeout, initiate leader election | ||
| 68 | 2025.05.14 01:41:24.331934 [ 12 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1 | ||
| 69 | 2025.05.14 01:41:24.331948 [ 12 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done | ||
| 70 | 2025.05.14 01:41:24.331975 [ 12 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 3, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 71 | 2025.05.14 01:41:24.332024 [ 12 ] {} <Trace> RaftInstance: send req 3 -> 2, type pre_vote_request | ||
| 72 | 2025.05.14 01:41:24.332037 [ 12 ] {} <Debug> RaftInstance: socket 1bc06060 to ::1:9235 is not opened yet | ||
| 73 | 2025.05.14 01:41:24.332169 [ 12 ] {} <Trace> RaftInstance: send req 3 -> 1, type pre_vote_request | ||
| 74 | 2025.05.14 01:41:24.332183 [ 12 ] {} <Debug> RaftInstance: socket 1bc071e0 to ::1:9234 is not opened yet | ||
| 75 | 2025.05.14 01:41:24.332192 [ 12 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 76 | 2025.05.14 01:41:24.332204 [ 12 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 77 | 2025.05.14 01:41:24.333344 [ 18 ] {} <Information> RaftInstance: 1bc06060 connected to ::1:9235 (as a client) | ||
| 78 | 2025.05.14 01:41:24.333468 [ 18 ] {} <Information> RaftInstance: 1bc071e0 connected to ::1:9234 (as a client) | ||
| 79 | 2025.05.14 01:41:24.333779 [ 19 ] {} <Trace> RaftInstance: resp of req 3 -> 1, type pre_vote_request, OK | ||
| 80 | 2025.05.14 01:41:24.333802 [ 19 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 1 with Result=1, Term=0, NextIndex=1 | ||
| 81 | 2025.05.14 01:41:24.333785 [ 15 ] {} <Trace> RaftInstance: resp of req 3 -> 2, type pre_vote_request, OK | ||
| 82 | 2025.05.14 01:41:24.333821 [ 19 ] {} <Trace> RaftInstance: src: 1, dst: 3, resp->get_term(): 0 | ||
| 83 | 2025.05.14 01:41:24.333893 [ 19 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 1 (O), term 0, resp term 0, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 84 | 2025.05.14 01:41:24.333902 [ 19 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 | ||
| 85 | 2025.05.14 01:41:24.333914 [ 19 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote | ||
| 86 | 2025.05.14 01:41:24.334149 [ 19 ] {} <Information> RaftInstance: [VOTE INIT] my id 3, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1) | ||
| 87 | 2025.05.14 01:41:24.334169 [ 19 ] {} <Debug> RaftInstance: send request_vote_request to server 2 with term 1 | ||
| 88 | 2025.05.14 01:41:24.334176 [ 19 ] {} <Trace> RaftInstance: send req 3 -> 2, type request_vote_request | ||
| 89 | 2025.05.14 01:41:24.334236 [ 19 ] {} <Debug> RaftInstance: send request_vote_request to server 1 with term 1 | ||
| 90 | 2025.05.14 01:41:24.334244 [ 19 ] {} <Trace> RaftInstance: send req 3 -> 1, type request_vote_request | ||
| 91 | 2025.05.14 01:41:24.334288 [ 19 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 92 | 2025.05.14 01:41:24.334303 [ 19 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 93 | 2025.05.14 01:41:24.334361 [ 15 ] {} <Debug> RaftInstance: Receive a pre_vote_response message from peer 2 with Result=1, Term=0, NextIndex=1 | ||
| 94 | 2025.05.14 01:41:24.334418 [ 15 ] {} <Trace> RaftInstance: src: 2, dst: 3, resp->get_term(): 0 | ||
| 95 | 2025.05.14 01:41:24.334436 [ 15 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 0, resp term 0, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2 | ||
| 96 | 2025.05.14 01:41:24.334446 [ 15 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0 | ||
| 97 | 2025.05.14 01:41:24.334475 [ 15 ] {} <Information> RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing | ||
| 98 | 2025.05.14 01:41:24.334607 [ 15 ] {} <Trace> RaftInstance: resp of req 3 -> 1, type request_vote_request, OK | ||
| 99 | 2025.05.14 01:41:24.334615 [ 16 ] {} <Trace> RaftInstance: resp of req 3 -> 2, type request_vote_request, OK | ||
| 100 | 2025.05.14 01:41:24.334628 [ 15 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 1 with Result=0, Term=1, NextIndex=0 | ||
| 101 | 2025.05.14 01:41:24.334652 [ 15 ] {} <Trace> RaftInstance: src: 1, dst: 3, resp->get_term(): 1 | ||
| 102 | 2025.05.14 01:41:24.334662 [ 15 ] {} <Information> RaftInstance: [VOTE RESP] peer 1 (X), resp term 1, my role candidate, granted 1, responded 2, num voting members 3, quorum 2 | ||
| 103 | 2025.05.14 01:41:24.334686 [ 16 ] {} <Debug> RaftInstance: Receive a request_vote_response message from peer 2 with Result=0, Term=1, NextIndex=0 | ||
| 104 | 2025.05.14 01:41:24.334705 [ 16 ] {} <Trace> RaftInstance: src: 2, dst: 3, resp->get_term(): 1 | ||
| 105 | 2025.05.14 01:41:24.334716 [ 16 ] {} <Information> RaftInstance: [VOTE RESP] peer 2 (X), resp term 1, my role candidate, granted 1, responded 3, num voting members 3, quorum 2 | ||
| 106 | 2025.05.14 01:41:24.778246 [ 15 ] {} <Debug> RaftInstance: Receive a pre_vote_request message from 1 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=1 | ||
| 107 | 2025.05.14 01:41:24.778328 [ 15 ] {} <Information> RaftInstance: [PRE-VOTE REQ] my role candidate, from peer 1, log term: req 0 / mine 0 | ||
| 108 | last idx: req 0 / mine 0, term: req 1 / mine 1 | ||
| 109 | HB dead | ||
| 110 | 2025.05.14 01:41:24.778338 [ 15 ] {} <Information> RaftInstance: pre-vote decision: O (grant) | ||
| 111 | 2025.05.14 01:41:24.778358 [ 15 ] {} <Debug> RaftInstance: Response back a pre_vote_response message to 1 with Accepted=1, Term=1, NextIndex=1 | ||
| 112 | 2025.05.14 01:41:24.779354 [ 19 ] {} <Debug> RaftInstance: Receive a request_vote_request message from 1 with LastLogIndex=0, LastLogTerm 0, EntriesLength=0, CommitIndex=0 and Term=2 | ||
| 113 | 2025.05.14 01:41:24.779611 [ 19 ] {} <Debug> IDisk: Copying from LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.3-clickhouse-y2UGSY/snapshots) state to LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.3-clickhouse-y2UGSY/snapshots) state-OLD. | ||
| 114 | 2025.05.14 01:41:24.780087 [ 19 ] {} <Information> RaftInstance: [BECOME FOLLOWER] term 2 | ||
| 115 | 2025.05.14 01:41:24.780127 [ 19 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 116 | 2025.05.14 01:41:24.780161 [ 19 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 117 | 2025.05.14 01:41:24.780209 [ 19 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1 | ||
| 118 | 2025.05.14 01:41:24.780222 [ 19 ] {} <Information> RaftInstance: [VOTE REQ] my role follower, from peer 1, log term: req 0 / mine 0 | ||
| 119 | last idx: req 0 / mine 0, term: req 2 / mine 2 | ||
| 120 | priority: target 1 / mine 1, voted_for -1 | ||
| 121 | 2025.05.14 01:41:24.780232 [ 19 ] {} <Information> RaftInstance: decision: O (grant), voted_for 1, term 2 | ||
| 122 | 2025.05.14 01:41:24.780298 [ 19 ] {} <Debug> IDisk: Copying from LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.3-clickhouse-y2UGSY/snapshots) state to LocalStateFileDisk (path: /var/tmp/omicron_tmp/oximeter_db-11d8f51cb75f441f-test_ensure_schema_walks_through_multiple_steps_replicated.16475.3-clickhouse-y2UGSY/snapshots) state-OLD. | ||
| 123 | 2025.05.14 01:41:24.780456 [ 19 ] {} <Debug> RaftInstance: Response back a request_vote_response message to 1 with Accepted=1, Term=2, NextIndex=1 | ||
| 124 | 2025.05.14 01:41:24.781243 [ 15 ] {} <Debug> RaftInstance: Receive a append_entries_request message from 1 with LastLogIndex=0, LastLogTerm 0, EntriesLength=1, CommitIndex=0 and Term=2 | ||
| 125 | 2025.05.14 01:41:24.781326 [ 15 ] {} <Trace> RaftInstance: from peer 1, req type: 3, req term: 2, req l idx: 0 (1), req c idx: 0, my term: 2, my role: 1 | ||
| 126 | 2025.05.14 01:41:24.781338 [ 15 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 127 | 2025.05.14 01:41:24.781394 [ 15 ] {} <Trace> RaftInstance: [LOG OK] req log idx: 0, req log term: 0, my last log idx: 0, my log (0) term: 0 | ||
| 128 | 2025.05.14 01:41:24.781418 [ 15 ] {} <Debug> RaftInstance: [INIT] log_idx: 1, count: 0, log_store_->next_slot(): 1, req.log_entries().size(): 1 | ||
| 129 | 2025.05.14 01:41:24.781427 [ 15 ] {} <Debug> RaftInstance: [after SKIP] log_idx: 1, count: 0 | ||
| 130 | 2025.05.14 01:41:24.781436 [ 15 ] {} <Debug> RaftInstance: [after OVWR] log_idx: 1, count: 0 | ||
| 131 | 2025.05.14 01:41:24.781444 [ 15 ] {} <Trace> RaftInstance: append at 1, term 2, timestamp 0 | ||
| 132 | 2025.05.14 01:41:24.781879 [ 15 ] {} <Information> RaftInstance: receive a config change from leader at 1 | ||
| 133 | 2025.05.14 01:41:24.781919 [ 11 ] {} <Trace> RaftInstance: got log append completion notification: OK | ||
| 134 | 2025.05.14 01:41:24.781973 [ 15 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 0, quick_commit_index_ 0, state_->get_commit_idx() 0 | ||
| 135 | 2025.05.14 01:41:24.782011 [ 15 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 136 | 2025.05.14 01:41:24.782025 [ 11 ] {} <Trace> RaftInstance: got log append completion notification: OK | ||
| 137 | 2025.05.14 01:41:24.782039 [ 15 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 138 | 2025.05.14 01:41:24.782082 [ 15 ] {} <Trace> RaftInstance: batch size hint: 0 bytes | ||
| 139 | 2025.05.14 01:41:24.782090 [ 15 ] {} <Debug> RaftInstance: Response back a append_entries_response message to 1 with Accepted=1, Term=2, NextIndex=2 | ||
| 140 | 2025.05.14 01:41:24.782112 [ 1 ] {} <Debug> KeeperDispatcher: Quorum initialized | ||
| 141 | 2025.05.14 01:41:24.782413 [ 19 ] {} <Debug> RaftInstance: Receive a append_entries_request message from 1 with LastLogIndex=1, LastLogTerm 2, EntriesLength=0, CommitIndex=1 and Term=2 | ||
| 142 | 2025.05.14 01:41:24.782451 [ 19 ] {} <Trace> RaftInstance: from peer 1, req type: 3, req term: 2, req l idx: 1 (0), req c idx: 1, my term: 2, my role: 1 | ||
| 143 | 2025.05.14 01:41:24.782460 [ 19 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 144 | 2025.05.14 01:41:24.782460 [ 1 ] {} <Debug> KeeperDispatcher: Dispatcher initialized | ||
| 145 | 2025.05.14 01:41:24.782473 [ 19 ] {} <Trace> RaftInstance: [LOG OK] req log idx: 1, req log term: 2, my last log idx: 1, my log (1) term: 2 | ||
| 146 | 2025.05.14 01:41:24.782571 [ 19 ] {} <Debug> RaftInstance: trigger commit upto 1 | ||
| 147 | 2025.05.14 01:41:24.782582 [ 19 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 0 | ||
| 148 | 2025.05.14 01:41:24.782604 [ 19 ] {} <Trace> RaftInstance: commit_cv_ notify (local thread) | ||
| 149 | 2025.05.14 01:41:24.782636 [ 19 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 150 | 2025.05.14 01:41:24.782651 [ 19 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 151 | 2025.05.14 01:41:24.782679 [ 19 ] {} <Trace> RaftInstance: batch size hint: 0 bytes | ||
| 152 | 2025.05.14 01:41:24.782682 [ 20 ] {} <Trace> RaftInstance: commit_cv_ wake up | ||
| 153 | 2025.05.14 01:41:24.782697 [ 19 ] {} <Debug> RaftInstance: Response back a append_entries_response message to 1 with Accepted=1, Term=2, NextIndex=2 | ||
| 154 | 2025.05.14 01:41:24.782816 [ 20 ] {} <Debug> RaftInstance: commit upto 1, current idx 0 | ||
| 155 | 2025.05.14 01:41:24.782839 [ 20 ] {} <Trace> RaftInstance: commit upto 1, current idx 1 | ||
| 156 | 2025.05.14 01:41:24.782861 [ 20 ] {} <Information> RaftInstance: config at index 1 is committed, prev config log idx 0 | ||
| 157 | 2025.05.14 01:41:24.782957 [ 20 ] {} <Information> RaftInstance: new config log idx 1, prev log idx 0, cur config log idx 0, prev log idx 0 | ||
| 158 | 2025.05.14 01:41:24.782979 [ 20 ] {} <Debug> RaftInstance: system is reconfigured to have 3 servers, last config index: 0, this config index: 1 | ||
| 159 | 2025.05.14 01:41:24.783015 [ 20 ] {} <Information> RaftInstance: new configuration: log idx 1, prev log idx 0 | ||
| 160 | peer 1, DC ID 0, ::1:9234, voting member, 1 | ||
| 161 | peer 2, DC ID 0, ::1:9235, voting member, 1 | ||
| 162 | peer 3, DC ID 0, ::1:9236, voting member, 1 | ||
| 163 | my id: 3, leader: 1, term: 2 | ||
| 164 | 2025.05.14 01:41:24.783046 [ 20 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 165 | 2025.05.14 01:41:24.783066 [ 20 ] {} <Debug> RaftInstance: DONE: commit upto 1, current idx 1 | ||
| 166 | 2025.05.14 01:41:24.783082 [ 20 ] {} <Trace> RaftInstance: commit_cv_ sleep | ||
| 167 | 2025.05.14 01:41:24.783896 [ 1 ] {} <Information> Application: Listening for Keeper (tcp): [::]:9183 | ||
| 168 | 2025.05.14 01:41:24.784079 [ 1 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 178.27 KiB, peak 2.17 MiB, free memory in arenas 0.00 B, will set to 118.50 MiB (RSS), difference: 118.33 MiB | ||
| 169 | 2025.05.14 01:41:24.785611 [ 1 ] {} <Debug> ConfigReloader: Loading config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml' | ||
| 170 | 2025.05.14 01:41:24.792284 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml', performing update on configuration | ||
| 171 | 2025.05.14 01:41:24.792550 [ 1 ] {} <Trace> KeeperDispatcher: Configuration update triggered, but nothing changed for Raft | ||
| 172 | 2025.05.14 01:41:24.792694 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again. | ||
| 173 | 2025.05.14 01:41:24.792718 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/work/oxidecomputer/omicron/oximeter/db/src/configs/keeper_config.xml', performed update on configuration | ||
| 174 | 2025.05.14 01:41:24.796781 [ 1 ] {} <Information> Application: Ready for connections. | ||
| 175 | 2025.05.14 01:41:25.000928 [ 26 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 120.73 MiB, peak 127.20 MiB, free memory in arenas 0.00 B, will set to 118.73 MiB (RSS), difference: -2.00 MiB | ||
| 176 | 2025.05.14 01:41:25.281623 [ 15 ] {} <Debug> RaftInstance: Receive a append_entries_request message from 1 with LastLogIndex=1, LastLogTerm 2, EntriesLength=0, CommitIndex=1 and Term=2 | ||
| 177 | 2025.05.14 01:41:25.281677 [ 15 ] {} <Trace> RaftInstance: from peer 1, req type: 3, req term: 2, req l idx: 1 (0), req c idx: 1, my term: 2, my role: 1 | ||
| 178 | 2025.05.14 01:41:25.281708 [ 15 ] {} <Trace> RaftInstance: (update) new target priority: 1 | ||
| 179 | 2025.05.14 01:41:25.281723 [ 15 ] {} <Trace> RaftInstance: [LOG OK] req log idx: 1, req log term: 2, my last log idx: 1, my log (1) term: 2 | ||
| 180 | 2025.05.14 01:41:25.281733 [ 15 ] {} <Trace> RaftInstance: local log idx 1, target_commit_idx 1, quick_commit_index_ 1, state_->get_commit_idx() 1 | ||
| 181 | 2025.05.14 01:41:25.281747 [ 15 ] {} <Trace> RaftInstance: cancel existing timer | ||
| 182 | 2025.05.14 01:41:25.281785 [ 15 ] {} <Trace> RaftInstance: re-schedule election timer | ||
| 183 | 2025.05.14 01:41:25.281828 [ 15 ] {} <Trace> RaftInstance: batch size hint: 0 bytes | ||
| 184 | 2025.05.14 01:41:25.281837 [ 15 ] {} <Debug> RaftInstance: Response back a append_entries_response message to 1 with Accepted=1, Term=2, NextIndex=2 | ||
| 185 | 2025.05.14 01:41:25.321691 [ 18 ] {} <Error> RaftInstance: session 1 failed to read rpc header from socket ::1:51312 due to error 2, End of file, ref count 1 | ||
| 186 | 2025.05.14 01:41:25.321791 [ 18 ] {} <Trace> RaftInstance: asio rpc session destroyed: 1c90d9a0 | ||