2024.09.25 20:43:43.440397 [ 1 ] {} KeeperLogStore: No logs exists in /var/tmp/omicron_tmp/integration_test-13fe08fd3bd99210-test_raft_config_parsing.10491.1-oximeter-clickward-test/keeper-3/coordination/log. It's Ok if it's the first run of clickhouse-keeper. 2024.09.25 20:43:43.440961 [ 1 ] {} KeeperLogStore: Removing all changelogs 2024.09.25 20:43:43.443023 [ 1 ] {} KeeperStateManager: No state was read 2024.09.25 20:43:45.040923 [ 16 ] {} RaftInstance: session 1 failed to read rpc header from socket ::1:64538 due to error 2, End of file, ref count 1 2024.09.25 20:43:46.596062 [ 16 ] {} RaftInstance: Election timeout, initiate leader election 2024.09.25 20:43:46.597457 [ 19 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:46.604159 [ 13 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:47.103600 [ 16 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:47.654012 [ 16 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:48.255263 [ 19 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:48.750943 [ 18 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:48.906000 [ 15 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:49.656387 [ 14 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:50.406861 [ 15 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:51.157946 [ 14 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:51.908312 [ 18 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:52.659289 [ 17 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:53.411526 [ 13 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:54.161907 [ 13 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:54.912413 [ 12 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:55.663424 [ 13 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:56.414634 [ 12 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:57.165755 [ 19 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:57.917085 [ 15 ] {} RaftInstance: peer (1) response error: failed to connect to peer 1, ::1:39101, error 146, Connection refused 2024.09.25 20:43:58.668001 [ 13 ] {} RaftInstance: too verbose RPC error on peer (1), will suppress it from now 2024.09.25 20:44:00.919519 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:44:03.171255 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:44:05.424217 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:44:08.427727 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:44:08.755416 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1078 ms ago 2024.09.25 20:44:08.757157 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1080 ms ago 2024.09.25 20:44:09.178271 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:44:09.928984 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:44:10.679535 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:44:12.181062 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:44:12.931871 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:44:13.682301 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:44:14.433197 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:44:15.183875 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:44:15.934943 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:44:16.685724 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 20:44:17.436860 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6006 ms ago 2024.09.25 20:44:18.187770 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6757 ms ago 2024.09.25 20:44:18.755385 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7324 ms ago 2024.09.25 20:44:18.757007 [ 14 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:45:38.755427 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 97 warnings, 227 ms, 1 times 2024.09.25 20:45:46.786803 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:46:04.056187 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:46:04.807293 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:46:05.557988 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:46:06.308557 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:46:07.059349 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:46:07.810098 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:46:08.560249 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 20:46:08.755363 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5449 ms ago 2024.09.25 20:46:08.757089 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5451 ms ago 2024.09.25 20:46:09.311381 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:46:16.068333 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:46:26.581460 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:46:31.090227 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:46:31.841139 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:46:32.591798 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:46:33.342672 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 20:46:34.093253 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:46:34.844366 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:46:35.594988 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 20:46:36.345314 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:46:37.095802 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6756 ms ago 2024.09.25 20:46:37.846936 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7507 ms ago 2024.09.25 20:46:38.597474 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8257 ms ago 2024.09.25 20:46:38.755397 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8415 ms ago 2024.09.25 20:46:38.757007 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8417 ms ago 2024.09.25 20:46:39.348240 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9008 ms ago 2024.09.25 20:46:40.099018 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9759 ms ago 2024.09.25 20:46:40.849166 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10509 ms ago 2024.09.25 20:46:41.599797 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11260 ms ago 2024.09.25 20:46:42.350700 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12011 ms ago 2024.09.25 20:46:43.101342 [ 15 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:49:28.755514 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 194 warnings, 476 ms, 2 times 2024.09.25 20:49:31.281673 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:49:32.032694 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:49:32.783155 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:49:39.541448 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 785 ms ago 2024.09.25 20:49:40.291778 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1536 ms ago 2024.09.25 20:49:41.042620 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2287 ms ago 2024.09.25 20:49:41.793126 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3037 ms ago 2024.09.25 20:49:42.544062 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3788 ms ago 2024.09.25 20:49:43.294790 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4539 ms ago 2024.09.25 20:49:44.045699 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5290 ms ago 2024.09.25 20:49:44.796405 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6040 ms ago 2024.09.25 20:49:45.547555 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6791 ms ago 2024.09.25 20:49:46.298236 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7542 ms ago 2024.09.25 20:49:47.048885 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8293 ms ago 2024.09.25 20:49:47.799591 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9044 ms ago 2024.09.25 20:49:48.550475 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9794 ms ago 2024.09.25 20:49:48.755420 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9999 ms ago 2024.09.25 20:49:48.757008 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10001 ms ago 2024.09.25 20:49:49.301249 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10545 ms ago 2024.09.25 20:49:50.052073 [ 15 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:50:38.757249 [ 15 ] {} RaftInstance: recovered from long pause to peer 1, 64 warnings, 1 ms, 3 times 2024.09.25 20:50:43.358211 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:50:44.859285 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:50:45.610248 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:50:46.361103 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:50:50.866261 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:50:51.617001 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:50:52.367576 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:50:53.118597 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:50:53.868793 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:50:54.619851 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:50:55.370432 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 20:50:56.121226 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:50:59.124391 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:50:59.875062 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:51:00.625745 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:51:01.376768 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 20:51:02.127481 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3754 ms ago 2024.09.25 20:51:02.877894 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:51:03.628688 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 20:51:04.379198 [ 18 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:52:00.683764 [ 13 ] {} RaftInstance: connection to peer 1 is not active long time: 25523 ms, force re-connect 2024.09.25 20:52:08.757083 [ 19 ] {} RaftInstance: recovered from long pause to peer 1, 103 warnings, 1 ms, 4 times 2024.09.25 20:52:10.443630 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:52:11.194715 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:52:11.945262 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:52:13.446701 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:52:15.699467 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:52:16.449875 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:52:17.200776 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:52:17.951152 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:52:18.701876 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:52:18.755379 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3806 ms ago 2024.09.25 20:52:18.757210 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3808 ms ago 2024.09.25 20:52:19.452827 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:52:20.202952 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 20:52:20.953749 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:52:21.704683 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6756 ms ago 2024.09.25 20:52:22.455167 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7506 ms ago 2024.09.25 20:52:23.206038 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8257 ms ago 2024.09.25 20:52:23.956675 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9008 ms ago 2024.09.25 20:52:24.707429 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9758 ms ago 2024.09.25 20:52:25.457799 [ 19 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:52:26.689215 [ 19 ] {} RaftInstance: [EDGE CASE] RPC for 1 has been reset before returning error: current 1d87a550 (274), from parameter 1d87c810 (266) 2024.09.25 20:52:28.755508 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 23 warnings, 294 ms, 5 times 2024.09.25 20:52:30.713431 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:52:31.464078 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:52:32.215265 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:52:32.965928 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 20:52:33.716598 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3754 ms ago 2024.09.25 20:52:34.467381 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:52:35.218084 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 20:52:35.968769 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6006 ms ago 2024.09.25 20:52:36.719892 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6757 ms ago 2024.09.25 20:52:37.470349 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7507 ms ago 2024.09.25 20:52:38.220842 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8258 ms ago 2024.09.25 20:52:38.755597 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8793 ms ago 2024.09.25 20:52:38.757330 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8794 ms ago 2024.09.25 20:52:38.971506 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9009 ms ago 2024.09.25 20:52:39.722218 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9759 ms ago 2024.09.25 20:52:40.472709 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10510 ms ago 2024.09.25 20:52:41.223579 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11261 ms ago 2024.09.25 20:52:41.973983 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12011 ms ago 2024.09.25 20:52:42.724812 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12762 ms ago 2024.09.25 20:52:43.475109 [ 17 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:52:55.486187 [ 19 ] {} RaftInstance: connection to peer 1 is not active long time: 25523 ms, force re-connect 2024.09.25 20:53:18.757078 [ 13 ] {} RaftInstance: recovered from long pause to peer 1, 64 warnings, 1 ms, 6 times 2024.09.25 20:53:22.511143 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:53:26.265131 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:53:27.015759 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:53:27.766168 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:53:28.517040 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:53:28.756177 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3241 ms ago 2024.09.25 20:53:28.757945 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3243 ms ago 2024.09.25 20:53:29.267982 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:53:30.019000 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:53:30.769462 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 20:53:31.520345 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:53:32.270848 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6756 ms ago 2024.09.25 20:53:33.021802 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7507 ms ago 2024.09.25 20:53:33.772787 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8258 ms ago 2024.09.25 20:53:34.523557 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9008 ms ago 2024.09.25 20:53:35.274007 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9759 ms ago 2024.09.25 20:53:36.024897 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10510 ms ago 2024.09.25 20:53:36.775450 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11260 ms ago 2024.09.25 20:53:37.526515 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12011 ms ago 2024.09.25 20:53:38.276944 [ 18 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:53:58.756160 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 32 warnings, 205 ms, 7 times 2024.09.25 20:54:00.802458 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:54:06.809870 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:54:10.564337 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:54:11.314707 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:54:12.065553 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:54:12.419231 [ 19 ] {} RaftInstance: [EDGE CASE] RPC for 1 has been reset before returning error: current 1d87a550 (322), from parameter 1d87c810 (277) 2024.09.25 20:54:13.567508 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:54:16.570484 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:54:17.321134 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:54:18.071797 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:54:22.577177 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:54:26.332549 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:54:27.834511 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:54:28.585489 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:54:28.756173 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1672 ms ago 2024.09.25 20:54:28.757992 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1674 ms ago 2024.09.25 20:54:29.336175 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:54:30.837347 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:54:31.588072 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:54:32.338545 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:54:33.089275 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:54:33.839661 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3752 ms ago 2024.09.25 20:54:34.590633 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 20:54:35.341375 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 20:54:36.092081 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:54:36.842337 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6755 ms ago 2024.09.25 20:54:37.593080 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7506 ms ago 2024.09.25 20:54:38.343494 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8256 ms ago 2024.09.25 20:54:38.756149 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8669 ms ago 2024.09.25 20:54:38.757992 [ 13 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:54:55.610090 [ 12 ] {} RaftInstance: connection to peer 1 is not active long time: 25523 ms, force re-connect 2024.09.25 20:55:28.756180 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 71 warnings, 108 ms, 8 times 2024.09.25 20:55:30.900088 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:55:33.903599 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:55:34.654629 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:55:35.404937 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:55:36.907301 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:55:37.658037 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1502 ms ago 2024.09.25 20:55:38.408533 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:55:38.756193 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2600 ms ago 2024.09.25 20:55:38.758502 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2602 ms ago 2024.09.25 20:55:41.411857 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:55:42.913661 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:55:51.173257 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 20:55:51.923642 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:55:52.674351 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:55:54.175683 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:55:54.926092 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:55:55.676713 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:55:56.427209 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:55:57.177901 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:55:57.928808 [ 19 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:57:18.756252 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 85 warnings, 491 ms, 9 times 2024.09.25 20:57:25.024430 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:57:25.775105 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:57:26.525504 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:57:27.276471 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:57:28.027139 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:57:28.756171 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4482 ms ago 2024.09.25 20:57:28.757865 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4484 ms ago 2024.09.25 20:57:28.778255 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:57:29.529117 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 20:57:30.279787 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:57:32.531040 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:57:34.032729 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:57:34.783556 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:57:35.534107 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:57:36.285141 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 20:57:37.035567 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:57:37.786406 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:57:38.537056 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 20:57:38.756161 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5474 ms ago 2024.09.25 20:57:38.757869 [ 18 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:57:48.756297 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 30 warnings, 460 ms, 10 times 2024.09.25 20:57:55.053180 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:57:55.803843 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:57:56.554177 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:58:03.312334 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:58:04.062646 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:58:04.813437 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:58:05.563835 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:58:06.314556 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:58:07.065067 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 20:58:07.815966 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 20:58:08.566624 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:58:08.756188 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6194 ms ago 2024.09.25 20:58:08.758102 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6196 ms ago 2024.09.25 20:58:09.317360 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6756 ms ago 2024.09.25 20:58:10.067960 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7506 ms ago 2024.09.25 20:58:10.818664 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8257 ms ago 2024.09.25 20:58:11.569080 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9007 ms ago 2024.09.25 20:58:12.319832 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9758 ms ago 2024.09.25 20:58:12.659216 [ 17 ] {} RaftInstance: [EDGE CASE] RPC for 1 has been reset before returning error: current 1d87c140 (464), from parameter 1d87af90 (337) 2024.09.25 20:58:13.070056 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10508 ms ago 2024.09.25 20:58:13.820991 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11259 ms ago 2024.09.25 20:58:14.571363 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12010 ms ago 2024.09.25 20:58:15.322197 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12760 ms ago 2024.09.25 20:58:19.827147 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:58:24.333196 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:58:25.083911 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:58:25.834678 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:58:28.756216 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 669 ms ago 2024.09.25 20:58:28.757860 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 671 ms ago 2024.09.25 20:58:28.837322 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:58:33.342156 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:58:34.093270 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:58:34.844176 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:58:36.346082 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:58:37.096490 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:58:37.847214 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:58:38.597777 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:58:38.756226 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3160 ms ago 2024.09.25 20:58:38.757944 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3162 ms ago 2024.09.25 20:58:39.348261 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3752 ms ago 2024.09.25 20:58:40.098738 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 20:58:40.849532 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 20:58:41.600112 [ 18 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 20:59:18.756303 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 54 warnings, 364 ms, 1 times 2024.09.25 20:59:21.395073 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:59:22.896622 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:59:23.647140 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1500 ms ago 2024.09.25 20:59:24.398727 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:59:27.402441 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:59:28.756222 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 602 ms ago 2024.09.25 20:59:28.758006 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 604 ms ago 2024.09.25 20:59:28.903444 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:59:29.654190 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1500 ms ago 2024.09.25 20:59:30.405103 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 20:59:31.907430 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:59:32.657994 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 20:59:33.408605 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 20:59:34.159051 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 20:59:34.909845 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 20:59:35.660608 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 20:59:36.411524 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 20:59:37.162001 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 20:59:46.923343 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 20:59:47.674183 [ 12 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:00:08.756257 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 39 warnings, 60 ms, 2 times 2024.09.25 21:00:11.699198 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:00:13.200279 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:00:13.951033 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:00:14.701198 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:00:15.451996 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:00:19.206260 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:00:19.956920 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:00:20.707667 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:00:21.458501 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 21:00:22.209403 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:00:22.960045 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 21:00:23.710521 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 21:00:24.461426 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 21:00:26.714741 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:00:27.465458 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:00:28.215845 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:00:28.756212 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2792 ms ago 2024.09.25 21:00:28.757934 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2794 ms ago 2024.09.25 21:00:28.966458 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:00:31.217829 [ 17 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:00:48.756312 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 30 warnings, 269 ms, 3 times 2024.09.25 21:00:50.737732 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:00:52.239848 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:00:52.990211 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:00:53.740944 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:00:54.491294 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:00:59.748273 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 991 ms ago 2024.09.25 21:01:00.498934 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1742 ms ago 2024.09.25 21:01:01.249576 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2493 ms ago 2024.09.25 21:01:02.000639 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3244 ms ago 2024.09.25 21:01:02.751629 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3995 ms ago 2024.09.25 21:01:03.502245 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4745 ms ago 2024.09.25 21:01:04.253139 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5496 ms ago 2024.09.25 21:01:05.004042 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6247 ms ago 2024.09.25 21:01:05.754869 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6998 ms ago 2024.09.25 21:01:06.505409 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7749 ms ago 2024.09.25 21:01:07.256219 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8499 ms ago 2024.09.25 21:01:08.006575 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9250 ms ago 2024.09.25 21:01:08.756257 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9999 ms ago 2024.09.25 21:01:08.757400 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10001 ms ago 2024.09.25 21:01:08.758274 [ 16 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:01:58.756293 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 50 warnings, 440 ms, 4 times 2024.09.25 21:01:59.817255 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:02:00.567979 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:02:01.319068 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:02:07.326189 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:02:08.756251 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 678 ms ago 2024.09.25 21:02:08.758046 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 680 ms ago 2024.09.25 21:02:08.827473 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:02:09.577702 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1500 ms ago 2024.09.25 21:02:10.328618 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:02:11.078919 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3001 ms ago 2024.09.25 21:02:14.082730 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:02:14.834245 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1502 ms ago 2024.09.25 21:02:15.584939 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2253 ms ago 2024.09.25 21:02:20.089229 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:02:20.840098 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:02:21.590494 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:02:23.092325 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:02:23.843122 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:02:24.593656 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:02:29.099261 [ 14 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:02:56.874737 [ 18 ] {} RaftInstance: connection to peer 1 is not active long time: 25523 ms, force re-connect 2024.09.25 21:03:22.719207 [ 16 ] {} RaftInstance: [EDGE CASE] RPC for 1 has been reset before returning error: current 1d874d10 (650), from parameter 1d87c810 (638) 2024.09.25 21:03:38.757887 [ 13 ] {} RaftInstance: recovered from long pause to peer 1, 105 warnings, 1 ms, 5 times 2024.09.25 21:03:41.168915 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:03:42.670684 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:03:43.421372 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:03:44.172037 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:03:44.923044 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 21:03:56.937415 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:04:01.441820 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:04:05.947339 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:04:06.697806 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:04:07.448539 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:04:08.198776 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:04:08.756296 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3559 ms ago 2024.09.25 21:04:08.758037 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3561 ms ago 2024.09.25 21:04:08.949558 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:04:09.700065 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 21:04:10.450810 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 21:04:11.201178 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6004 ms ago 2024.09.25 21:04:23.214329 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:04:23.965164 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:04:24.715616 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:04:25.466308 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:04:26.216887 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:04:26.967652 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 21:04:27.718486 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 21:04:28.468624 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6004 ms ago 2024.09.25 21:04:28.756382 [ 21 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:04:38.758131 [ 17 ] {} RaftInstance: recovered from long pause to peer 1, 25 warnings, 1 ms, 6 times 2024.09.25 21:04:44.236316 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:04:46.489160 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:04:47.239505 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:04:47.990368 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:04:48.740976 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:04:48.756361 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3018 ms ago 2024.09.25 21:04:48.757924 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3019 ms ago 2024.09.25 21:04:49.491880 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:04:50.242640 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 21:04:50.993570 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 21:04:51.743780 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 21:04:55.498029 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:04:56.248224 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:04:56.999206 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:04:57.749762 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:04:58.500549 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:04:58.756289 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4009 ms ago 2024.09.25 21:04:58.757872 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4010 ms ago 2024.09.25 21:04:59.251098 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 21:05:00.001823 [ 15 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:05:20.271340 [ 18 ] {} RaftInstance: connection to peer 1 is not active long time: 25524 ms, force re-connect 2024.09.25 21:05:28.756376 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 55 warnings, 226 ms, 7 times 2024.09.25 21:05:30.031336 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:05:33.786535 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:05:34.537422 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:05:35.288234 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:05:36.039136 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 21:05:36.789720 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3754 ms ago 2024.09.25 21:05:37.540458 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 21:05:38.291133 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 21:05:38.756329 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5720 ms ago 2024.09.25 21:05:38.758046 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5722 ms ago 2024.09.25 21:05:39.041715 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6006 ms ago 2024.09.25 21:05:39.792654 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6757 ms ago 2024.09.25 21:05:40.543441 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7507 ms ago 2024.09.25 21:05:41.293775 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8258 ms ago 2024.09.25 21:05:42.044780 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9009 ms ago 2024.09.25 21:05:42.795323 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9759 ms ago 2024.09.25 21:05:43.545965 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10510 ms ago 2024.09.25 21:05:44.296163 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11260 ms ago 2024.09.25 21:05:45.046789 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12011 ms ago 2024.09.25 21:05:45.797223 [ 12 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:05:46.739204 [ 14 ] {} RaftInstance: [EDGE CASE] RPC for 1 has been reset before returning error: current 1d878300 (734), from parameter 1d87af90 (722) 2024.09.25 21:06:42.854428 [ 15 ] {} RaftInstance: connection to peer 1 is not active long time: 25524 ms, force re-connect 2024.09.25 21:06:43.279205 [ 18 ] {} RaftInstance: [EDGE CASE] RPC for 1 has been reset before returning error: current 1d88f7e0 (755), from parameter 1d87af90 (754) 2024.09.25 21:06:58.756377 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 92 warnings, 132 ms, 8 times 2024.09.25 21:07:01.626212 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:07:02.376521 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1500 ms ago 2024.09.25 21:07:03.127420 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:07:03.877652 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3001 ms ago 2024.09.25 21:07:04.628572 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3752 ms ago 2024.09.25 21:07:05.378850 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 21:07:06.129474 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5253 ms ago 2024.09.25 21:07:06.879869 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6004 ms ago 2024.09.25 21:07:07.631106 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6755 ms ago 2024.09.25 21:07:08.381606 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7505 ms ago 2024.09.25 21:07:08.756343 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7880 ms ago 2024.09.25 21:07:08.758096 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7882 ms ago 2024.09.25 21:07:09.132283 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8256 ms ago 2024.09.25 21:07:09.882744 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9007 ms ago 2024.09.25 21:07:10.633657 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9757 ms ago 2024.09.25 21:07:11.384127 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10508 ms ago 2024.09.25 21:07:12.135336 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11259 ms ago 2024.09.25 21:07:12.885890 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12010 ms ago 2024.09.25 21:07:13.636934 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12761 ms ago 2024.09.25 21:07:15.138902 [ 17 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:07:18.758324 [ 19 ] {} RaftInstance: recovered from long pause to peer 1, 21 warnings, 1 ms, 9 times 2024.09.25 21:07:33.910552 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:07:34.661416 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:07:35.411990 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:07:39.917914 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:07:40.668823 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:07:41.419852 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:07:45.175171 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:07:48.178211 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:07:48.756278 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1328 ms ago 2024.09.25 21:07:48.757914 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1330 ms ago 2024.09.25 21:07:48.929125 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:07:49.680084 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:07:54.185670 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:07:54.936198 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:07:55.688252 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2253 ms ago 2024.09.25 21:07:56.438623 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 21:07:57.189409 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3754 ms ago 2024.09.25 21:07:57.939900 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4505 ms ago 2024.09.25 21:07:58.690788 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5256 ms ago 2024.09.25 21:07:58.756337 [ 21 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:08:18.757769 [ 12 ] {} RaftInstance: recovered from long pause to peer 1, 41 warnings, 1 ms, 10 times 2024.09.25 21:10:40.115611 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:10:46.122970 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:10:47.624474 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:10:48.375112 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:10:48.756398 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1883 ms ago 2024.09.25 21:10:48.758177 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1884 ms ago 2024.09.25 21:10:49.125781 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:10:50.627869 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:10:55.882557 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:10:56.633431 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:10:57.384135 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:10:58.134801 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 21:10:58.756290 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3624 ms ago 2024.09.25 21:10:58.757947 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3626 ms ago 2024.09.25 21:10:58.885436 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:10:59.636552 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 21:11:00.386829 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 21:11:01.137812 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6006 ms ago 2024.09.25 21:11:07.145423 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:11:07.895660 [ 17 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:11:31.919249 [ 13 ] {} RaftInstance: connection to peer 1 is not active long time: 25524 ms, force re-connect 2024.09.25 21:11:57.579284 [ 15 ] {} RaftInstance: [EDGE CASE] RPC for 1 has been reset before returning error: current 1d87a550 (1047), from parameter 1d87ccb0 (1031) 2024.09.25 21:12:18.756407 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 93 warnings, 287 ms, 1 times 2024.09.25 21:12:26.730094 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:12:31.987405 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:12:32.738319 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:12:33.488798 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:12:37.993439 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:12:38.744494 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:12:38.756349 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1513 ms ago 2024.09.25 21:12:38.758203 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1515 ms ago 2024.09.25 21:12:39.495384 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:12:40.997531 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:12:47.006377 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:12:47.757107 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:12:48.507607 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:12:48.756425 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2500 ms ago 2024.09.25 21:12:48.758235 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2502 ms ago 2024.09.25 21:12:49.258626 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:12:53.013461 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:12:56.017656 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:13:04.277264 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:13:05.027682 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:13:05.778436 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:13:06.529034 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:13:07.280127 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:13:08.030634 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 21:13:08.756380 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5230 ms ago 2024.09.25 21:13:08.758050 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5231 ms ago 2024.09.25 21:13:08.781791 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 21:13:09.532396 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6006 ms ago 2024.09.25 21:13:12.536374 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:13:14.038425 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:13:14.789365 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:13:15.539662 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:13:16.290390 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:13:19.294142 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:13:20.044762 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:13:20.795824 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:13:21.546140 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:13:22.296929 [ 19 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:13:48.756520 [ 21 ] {} RaftInstance: recovered from long pause to peer 1, 38 warnings, 168 ms, 2 times 2024.09.25 21:13:50.839707 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:13:54.594132 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:13:55.344783 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:13:56.095010 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:14:02.103088 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:14:03.604716 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:14:04.355600 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:14:05.106424 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:14:06.607726 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:14:07.358682 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:14:08.108872 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:14:11.112647 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:14:12.614334 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:14:13.365155 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:14:14.115471 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:14:14.866597 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:14:15.616811 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:14:16.367826 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 21:14:17.118709 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 21:14:17.869491 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 21:14:18.620253 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6756 ms ago 2024.09.25 21:14:18.756381 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6892 ms ago 2024.09.25 21:14:18.757977 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6894 ms ago 2024.09.25 21:14:19.371321 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7507 ms ago 2024.09.25 21:14:20.121491 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8257 ms ago 2024.09.25 21:14:20.872289 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9008 ms ago 2024.09.25 21:14:21.622849 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9759 ms ago 2024.09.25 21:14:22.373694 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10510 ms ago 2024.09.25 21:14:23.124702 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11261 ms ago 2024.09.25 21:14:23.875438 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12011 ms ago 2024.09.25 21:14:24.626097 [ 19 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:14:38.758176 [ 17 ] {} RaftInstance: recovered from long pause to peer 1, 26 warnings, 1 ms, 3 times 2024.09.25 21:14:41.892976 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:14:42.643927 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1502 ms ago 2024.09.25 21:14:43.394178 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:14:44.144920 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 21:14:44.895872 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:14:45.646697 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4504 ms ago 2024.09.25 21:14:46.397547 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5255 ms ago 2024.09.25 21:14:47.148361 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6006 ms ago 2024.09.25 21:14:47.899331 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6757 ms ago 2024.09.25 21:14:48.649721 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7507 ms ago 2024.09.25 21:14:48.756453 [ 21 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7614 ms ago 2024.09.25 21:14:48.758056 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 7616 ms ago 2024.09.25 21:14:49.400007 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 8258 ms ago 2024.09.25 21:14:50.150315 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9008 ms ago 2024.09.25 21:14:50.901200 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 9759 ms ago 2024.09.25 21:14:51.652443 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 10510 ms ago 2024.09.25 21:14:52.403752 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 11261 ms ago 2024.09.25 21:14:53.154440 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 12012 ms ago 2024.09.25 21:14:56.909442 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:14:57.662120 [ 18 ] {} RaftInstance: long pause warning to 1 is too verbose, will suppress it from now 2024.09.25 21:15:38.762462 [ 12 ] {} RaftInstance: recovered from long pause to peer 1, 46 warnings, 1 ms, 4 times 2024.09.25 21:15:42.717012 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:15:43.467764 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:15:44.218237 [ 14 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2251 ms ago 2024.09.25 21:15:44.968896 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3002 ms ago 2024.09.25 21:15:45.719548 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3753 ms ago 2024.09.25 21:15:46.470419 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 4503 ms ago 2024.09.25 21:15:47.220746 [ 12 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 5254 ms ago 2024.09.25 21:15:47.971468 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 6005 ms ago 2024.09.25 21:15:51.726231 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:15:52.477400 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:15:53.227701 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:15:57.732808 [ 19 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:16:00.736250 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:16:01.487325 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 1501 ms ago 2024.09.25 21:16:02.238233 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 2252 ms ago 2024.09.25 21:16:02.988999 [ 17 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 3003 ms ago 2024.09.25 21:16:05.241798 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:16:10.497183 [ 18 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:16:11.999518 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 751 ms ago 2024.09.25 21:16:15.003099 [ 15 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:16:54.798955 [ 16 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 750 ms ago 2024.09.25 21:17:19.574483 [ 13 ] {} RaftInstance: skipped sending msg to 1 too long time, last msg sent 817 ms ago