starting January 2, 2024 at 10:30:13 PM UTC Creating downstairs regions Creating region directory at: /var/tmp/dsc/region Creating dsc directory at: /tmp/dsc Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.5901799 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.08167245 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.083514474 Region set with 3 regions was created Starting downstairs Using existing output directory "/tmp/dsc" start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] reports Starting [8820][1] reports Starting [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1175) [8810][0] reports Running Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1176) [8820][1] reports Running Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1177) [8830][2] reports Running Jan 02 22:30:14.709 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Running initial fill {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.815748847Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"6c4e859e6805a0246bf9e520f0708beaee75edc8\",\n git_commit_timestamp: \"2024-01-02T22:01:24.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.73.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"cc66ad468955717ab92600c770da8c1601a4ff33\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.817599939Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.817629622Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181} {"msg":"Crucible ba9a8098-f011-4e69-8ec9-5a3b11c2665d has session id: d9c167c2-6cd4-4d09-8255-7477fa43a5db","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.817651145Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Upstairs opts: Upstairs UUID: ba9a8098-f011-4e69-8ec9-5a3b11c2665d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.818215983Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Crucible stats registered with UUID: ba9a8098-f011-4e69-8ec9-5a3b11c2665d","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.818246401Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} Crucible runtime is spawned The guest has requested activation with gen:1 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.821406712Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.821542036Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.821577583Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ba9a8098-f011-4e69-8ec9-5a3b11c2665d active request set","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.821717961Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.821830969Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.821950371Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.821992377Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.822638828Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.822697112Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.822741765Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.828589685Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.828631407Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.828656141Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.828678919Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.828710324Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.828752794Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.833050707Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.833173449Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.833278252Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835296078Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835355205Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835401792Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835439563Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835481125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835522048Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835560993Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835628776Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835672444Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.835715106Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836698619Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836779223Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836808533Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836837704Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836861505Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836884591Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836908568Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836930394Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.836952647Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.83697435Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.837931328Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"":"downstairs","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"No downstairs repair required","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.837961441Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.837982616Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"0","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.838005411Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"1","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.838027905ZThe guest has finished waiting for activation with:1 ","hostname":Activate successful "Wait for a query_work_queue command to finish before sending IO ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"client":"2","":"downstairs","session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"All required repair work is completed","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.838058708Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Set Downstairs and Upstairs active after repairs","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.838079235Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"ba9a8098-f011-4e69-8ec9-5a3b11c2665d is now active with session: d9c167c2-6cd4-4d09-8255-7477fa43a5db","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.838105703Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} {"msg":"Set Active after no repair","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:19.838126371Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1181,"session_id":"d9c167c2-6cd4-4d09-8255-7477fa43a5db"} Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..5000 range:false) Wrote out file "alan" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:7 States: Active Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 99 Acked 1098 Read 100 Done Done Done false 100 Acked 1099 Read 100 Done Done Done false 101 Acked 1100 Read 100 Done Done Done false 102 Acked 1101 Read 100 Done Done Done false 103 Acked 1102 Read 100 Done Done Done false 104 Acked 1103 Read 100 Done Done Done false 105 Acked 1104 Read 100 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 7 7 7 21 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1097 1097 1097 Downstairs last five completed: 1097 1096 1095 1094 1093 Upstairs last five completed: 105 104 103 102 101 CLIENT: Up:0 ds:7 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1105 1104 1103 1102 1101 Upstairs last five completed: 106 105 104 103 102 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program Replay loop starts now January 2, 2024 at 10:30:26 PM UTC {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.156590942Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"6c4e859e6805a0246bf9e520f0708beaee75edc8\",\n git_commit_timestamp: \"2024-01-02T22:01:24.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.73.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"cc66ad468955717ab92600c770da8c1601a4ff33\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.15697232Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.156997988Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185} {"msg":"Crucible 2fcac7ae-e871-4d24-be00-c8291044a5ca has session id: 836679d4-4c63-42d3-928f-dc832a5b8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157019724Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Upstairs opts: Upstairs UUID: 2fcac7ae-e871-4d24-be00-c8291044a5ca, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157053383Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Crucible stats registered with UUID: 2fcac7ae-e871-4d24-be00-c8291044a5ca","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157076312Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Crucible runtime is spawned {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30The guest has requested activation with gen:2 ,"time":"2024-01-02T22:30:26.157356767Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157411005Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157441278Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"2fcac7ae-e871-4d24-be00-c8291044a5ca active request set","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157674124Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157774338Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.15782029Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157865798Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157909299Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157954182Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.157991413Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.158804901Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.158846741Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.158871314Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.15889401Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.1589167Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.158945031Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.160069031Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.160192791Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.160357943Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161454104Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161495429Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161520393Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.16168518Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161715525Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161738905Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161759297Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161781328Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161804466Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161827446Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161857166Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161881039Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161903836Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161927619Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161950448Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161973366Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.161996883Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162018723Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30The guest has finished waiting for activation with:2 ,"time":"Activate successful 2024-01-02T22:30:26.162040954Z"Wait for a query_work_queue command to finish before sending IO ,"hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Next flush: 3","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162068736Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162090363Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"No downstairs repair required","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.16211226Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.16213242Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162154242Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162180588Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"All required repair work is completed","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162203377Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Set Downstairs and Upstairs active after repairs","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162224018Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"2fcac7ae-e871-4d24-be00-c8291044a5ca is now active with session: 836679d4-4c63-42d3-928f-dc832a5b8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162245713Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Set Active after no repair","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.162265493Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576 Loading write count information from file "alan" Jan 02 22:30:26.253 INFO accepted connection, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Stop(2) stop 2 [8830] Got stop action so:false kr:false Jan 02 22:30:26.256 INFO request completed, latency_us: 1041, response_code: 204, uri: /stop/cid/2, method: POST, req_id: ee2c2222-8fd4-4f54-bd0b-d87961386486, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:26.257939155Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:26.258112702Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:26.258148964Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.258170869Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.258193482Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:26.258253908Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:30:26.258 INFO request completed, latency_us: 315, response_code: 200, uri: /state/cid/2, method: GET, req_id: 033b42e2-0ac6-4ad8-8ab2-943667fadec0, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:27.260335106Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:27.260574005Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:27.260609896Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:27.260637488Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:27.260657838Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:27.260750398Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:27.260847285Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:28.261759784Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:28.26182453Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:28.261851926Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:28.261878473Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:28.261898556Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:28.261939243Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:28.261984243Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:29.26319904Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:29.2632686Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:29.263297416Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:29.263324328Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:29.263344492Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:29.263376397Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:29.263650501Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:30:30.261 INFO request completed, latency_us: 259, response_code: 200, uri: /state/cid/2, method: GET, req_id: b0633fd5-8830-444e-b493-2eb5325e4ec3, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 4579 len 36864 data: 2 2 2 2 2 2 2 2 2 002/300 Read block 3010 len 16384 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:30.264938014Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:30.26503476Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:30.265072114Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:30.265099555Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:30.265119994Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 2 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:30.265149753Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:30.265172793Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 1326 len 4096 004/300 Read block 3720 len 8192 005/300 Read block 3228 len 4096 006/300 Write block 663 len 20480 data: 2 2 2 2 2 007/300 Read block 2473 len 4096 008/300 Read block 2304 len 28672 009/300 Write block 4248 len 40960 data: 2 2 2 2 2 2 2 2 2 2 010/300 Write block 1904 len 36864 data: 2 2 2 2 2 2 2 2 2 011/300 Write block 2413 len 8192 data: 2 2 012/300 Flush 013/300 Write block 3339 len 12288 data: 2 2 2 014/300 Read block 1041 len 36864 015/300 Read block 1716 len 32768 016/300 Write block 1870 len 36864 data: 2 2 2 2 2 2 2 2 2 017/300 Flush 018/300 Write block 3994 len 16384 data: 2 2 2 2 019/300 Write block 2506 len 12288 data: 2 2 2 020/300 Read block 2190 len 4096 021/300 Read block 1796 len 36864 022/300 Read block 994 len 16384 023/300 Write block 3132 len 12288 data: 2 2 2 024/300 Flush 025/300 Read block 741 len 16384 026/300 Write block 2375 len 12288 data: 2 2 2 027/300 Read block 1122 len 24576 028/300 Flush 029/300 Write block 3422 len 28672 data: 2 2 2 2 2 2 2 030/300 Read block 2708 len 40960 031/300 Read block 2117 len 20480 032/300 Read block 1663 len 16384 033/300 Read block 3421 len 36864 034/300 Flush 035/300 Read block 4278 len 32768 036/300 Write block 1064 len 28672 data: 2 2 2 2 2 2 2 037/300 Write block 3297 len 8192 data: 2 2 038/300 Write block 4839 len 24576 data: 2 2 2 2 2 2 039/300 Read block 3170 len 24576 040/300 Read block 4224 len 32768 041/300 Write block 2934 len 4096 data: 2 042/300 Read block 3841 len 32768 043/300 Write block 1794 len 8192 data: 2 2 044/300 Read block 3025 len 32768 045/300 Write block 2442 len 40960 data: 2 2 2 2 2 2 2 2 2 2 046/300 Read block 1565 len 16384 047/300 Write block 2077 len 12288 data: 2 2 2 048/300 Read block 4990 len 28672 049/300 Write block 392 len 16384 data: 2 2 2 2 050/300 Read block 1372 len 12288 051/300 Read block 1089 len 40960 052/300 Read block 2316 len 28672 053/300 Write block 586 len 20480 data: 2 2 2 2 2 054/300 Write block 4601 len 24576 data: 2 2 2 2 2 2 055/300 Write block 2047 len 8192 data: 2 2 056/300 Read block 417 len 24576 057/300 Read block 4478 len 40960 058/300 Read block 3977 len 4096 059/300 Write block 48 len 12288 data: 2 2 2 060/300 Write block 972 len 28672 data: 2 2 2 2 2 2 2 061/300 Read block 1920 len 36864 062/300 Write block 2989 len 12288 data: 2 2 2 063/300 Write block 2687 len 40960 data: 2 2 2 2 2 2 2 2 2 2 064/300 Write block 1499 len 12288 data: 2 2 2 065/300 Read block 1826 len 12288 066/300 Write block 2185 len 20480 data: 2 2 2 2 2 067/300 Read block 893 len 28672 068/300 Write block 474 len 24576 data: 2 2 2 2 2 2 069/300 Flush 070/300 Read block 2567 len 24576 071/300 Read block 3426 len 8192 072/300 Read block 3106 len 16384 073/300 Read block 3568 len 28672 074/300 Write block 4388 len 8192 data: 2 2 075/300 Read block 4545 len 40960 076/300 Write block 2332 len 24576 data: 2 2 2 2 2 2 077/300 Read block 980 len 40960 078/300 Read block 568 len 8192 079/300 Read block 936 len 8192 080/300 Write block 1714 len 36864 data: 2 2 2 2 2 2 2 2 2 081/300 Read block 376 len 28672 082/300 Write block 4609 len 32768 data: 2 2 2 2 2 2 2 2 083/300 Write block 3861 len 16384 data: 2 2 2 2 084/300 Write block 514 len 24576 data: 2 2 2 2 2 2 085/300 Read block 1490 len 20480 086/300 Read block 2562 len 12288 087/300 Write block 3561 len 8192 data: 2 2 088/300 Write block 2863 len 36864 data: 2 2 2 2 2 2 2 2 2 089/300 Write block 46 len 40960 data: 2 2 3 3 3 2 2 2 2 2 090/300 Read block 4263 len 12288 091/300 Write block 4528 len 4096 data: 2 092/300 Flush 093/300 Flush 094/300 Write block 4722 len 32768 data: 2 2 2 2 2 2 2 2 095/300 Read block 4576 len 28672 096/300 Read block 4954 len 36864 097/300 Write block 1890 len 4096 data: 2 098/300 Write block 1322 len 4096 data: 2 099/300 Read block 3577 len 40960 100/300 Read block 3362 len 20480 101/300 Write block 1421 len 20480 data: 2 2 2 2 2 102/300 Write block 1558 len 16384 data: 2 2 2 2 103/300 Write block 1676 len 24576 data: 2 2 2 2 2 2 104/300 Read block 4362 len 16384 105/300 Read block 308 len 16384 106/300 Write block 3723 len 36864 data: 2 2 2 2 2 2 2 2 2 107/300 Read block 638 len 32768 108/300 Read block 1256 len 12288 109/300 Write block 2029 len 40960 data: 2 2 2 2 2 2 2 2 2 2 110/300 Read block 2020 len 16384 111/300 Write block 4202 len 32768 data: 2 2 2 2 2 2 2 2 112/300 Read block 1791 len 8192 113/300 Flush 114/300 Read block 4830 len 8192 115/300 Flush 116/300 Write block 696 len 28672 data: 2 2 2 2 2 2 2 117/300 Read block 1182 len 12288 118/300 Read block 2233 len 36864 119/300 Write block 4925 len 16384 data: 2 2 2 2 120/300 Write block 4733 len 36864 data: 2 2 2 2 2 2 2 2 2 121/300 Write block 2505 len 32768 data: 2 3 3 3 2 2 2 2 122/300 Read block 3327 len 8192 123/300 Read block 242 len 28672 124/300 Read block 4641 len 4096 125/300 Read block 61 len 36864 126/300 Write block 516 len 12288 data: 3 3 3 127/300 Write block 4446 len 12288 data: 2 2 2 128/300 Write block 1389 len 20480 data: 2 2 2 2 2 129/300 Write block 3133 len 36864 data: 3 3 2 2 2 2 2 2 2 130/300 Write block 3832 len 28672 data: 2 2 2 2 2 2 2 131/300 Write block 3062 len 4096 data: 2 132/300 Read block 4741 len 24576 133/300 Write block 4836 len 28672 data: 2 2 2 3 3 3 3 134/300 Read block 4036 len 24576 135/300 Read block 4712 len 24576 136/300 Flush 137/300 Read block 836 len 20480 138/300 Read block 3060 len 8192 139/300 Write block 4399 len 16384 data: 2 2 2 2 140/300 Read block 4515 len 16384 141/300 Read block 1521 len 16384 142/300 Write block 2428 len 4096 data: 2 143/300 Flush 144/300 Read block 2016 len 24576 145/300 Read block 3966 len 32768 146/300 Write block 4437 len 12288 data: 2 2 2 147/300 Flush 148/300 Read block 111 len 12288 149/300 Read block 3281 len 4096 150/300 Read block 2127 len 20480 151/300 Read block 2532 len 4096 152/300 Read block 2327 len 40960 153/300 Read block 4692 len 28672 154/300 Write block 4766 len 32768 data: 2 2 2 2 2 2 2 2 155/300 Read block 4384 len 40960 156/300 Write block 1243 len 32768 data: 2 2 2 2 2 2 2 2 157/300 Flush 158/300 Write block 3716 len 8192 data: 2 2 159/300 Read block 4774 len 36864 160/300 Write block 1504 len 40960 data: 2 2 2 2 2 2 2 2 2 2 161/300 Read block 500 len 8192 162/300 Write block 160 len 12288 data: 2 2 2 163/300 Flush 164/300 Read block 1605 len 16384 165/300 Flush 166/300 Read block 3037 len 12288 167/300 Read block 4897 len 40960 168/300 Read block 1008 len 20480 169/300 Read block 4760 len 32768 170/300 Write block 3224 len 40960 data: 2 2 2 2 2 2 2 2 2 2 171/300 Write block 1548 len 8192 data: 2 2 172/300 Read block 3058 len 40960 173/300 Flush 174/300 Read block 1793 len 32768 175/300 Read block 3641 len 20480 176/300 Read block 4637 len 32768 177/300 Read block 4223 len 20480 178/300 Read block 4682 len 24576 179/300 Read block 2045 len 4096 180/300 Read block 3641 len 36864 181/300 Read block 4247 len 24576 182/300 Read block 1703 len 4096 183/300 Flush 184/300 Flush 185/300 Write block 3787 len 32768 data: 2 2 2 2 2 2 2 2 186/300 Read block 4645 len 4096 187/300 Write block 453 len 4096 data: 2 188/300 Read block 2990 len 20480 189/300 Read block 2875 len 32768 190/300 Flush 191/300 Write block 1344 len 20480 data: 2 2 2 2 2 192/300 Read block 2443 len 20480 193/300 Flush 194/300 Read block 2466 len 16384 195/300 Flush 196/300 Flush 197/300 Read block 2695 len 4096 198/300 Read block 2326 len 24576 199/300 Write block 2651 len 40960 data: 2 2 2 2 2 2 2 2 2 2 200/300 Write block 2036 len 36864 data: 3 3 3 2 2 2 2 2 2 201/300 Read block 1908 len 36864 202/300 Read block 551 len 32768 203/300 Read block 1260 len 16384 204/300 Read block 4406 len 4096 205/300 Write block 890 len 24576 data: 2 2 2 2 2 2 206/300 Flush 207/300 Read block 3200 len 40960 208/300 Write block 3882 len 28672 data: 2 2 2 2 2 2 2 209/300 Write block 38 len 16384 data: 2 2 2 2 210/300 Read block 3447 len 32768 211/300 Read block 2400 len 40960 212/300 Read block 1163 len 32768 213/300 Read block 3461 len 36864 214/300 Write block 4457 len 24576 data: 2 2 2 2 2 2 215/300 Read block 1353 len 20480 216/300 Write block 1669 len 32768 data: 2 2 2 2 2 2 2 3 217/300 Read block 4696 len 28672 218/300 Read block 4771 len 12288 219/300 Read block 1164 len 36864 220/300 Write block 2136 len 4096 data: 2 221/300 Write block 3050 len 40960 data: 2 2 2 2 2 2 2 2 2 2 222/300 Write block 4979 len 24576 data: 2 2 2 2 2 2 223/300 Read block 3830 len 24576 224/300 Read block 2425 len 8192 225/300 Write block 2227 len 16384 data: 2 2 2 2 226/300 Write block 1005 len 40960 data: 2 2 2 2 2 2 2 2 2 2 227/300 Write block 3960 len 20480 data: 2 2 2 2 2 228/300 Read block 1581 len 28672 229/300 Read block 1911 len 20480 230/300 Read block 2023 len 16384 231/300 Write block 2349 len 8192 data: 2 2 232/300 Write block 3957 len 24576 data: 2 2 2 3 3 3 233/300 Read block 2924 len 4096 234/300 Read block 242 len 20480 235/300 Write block 4341 len 32768 data: 2 2 2 2 2 2 2 2 236/300 Write block 3308 len 16384 data: 2 2 2 2 237/300 Read block 249 len 20480 238/300 Flush 239/300 Read block 743 len 20480 240/300 Flush 241/300 Flush 242/300 Read block 3562 len 20480 243/300 Read block 4201 len 28672 244/300 Write block 4423 len 8192 data: 2 2 245/300 Write block 3115 len 4096 data: 2 246/300 Write block 2961 len 20480 data: 2 2 2 2 2 247/300 Read block 2363 len 24576 248/300 Write block 1283 len 8192 data: 2 2 249/300 Read block 342 len 8192 250/300 Write block 4902 len 36864 data: 2 2 2 2 2 2 2 2 2 251/300 Write block 573 len 4096 data: 2 252/300 Write block 3134 len 40960 data: 4 3 3 3 3 3 3 3 2 2 253/300 Write block 1350 len 24576 data: 2 2 2 2 2 2 254/300 Write block 3757 len 8192 data: 2 2 255/300 Write block 4974 len 12288 data: 2 2 2 256/300 Read block 3296 len 36864 257/300 Read block 3860 len 20480 258/300 Read block 1304 len 8192 259/300 Flush 260/300 Write block 578 len 16384 data: 2 2 2 2 261/300 Flush 262/300 Read block 448 len 28672 263/300 Read block 1781 len 8192 264/300 Write block 3958 len 8192 data: 3 3 265/300 Read block 1782 len 12288 266/300 Flush 267/300 Read block 2659 len 24576 268/300 Read block 2658 len 20480 269/300 Write block 4603 len 20480 data: 3 3 3 3 2 270/300 Write block 4441 len 16384 data: 2 2 2 2 271/300 Write block 3316 len 20480 data: 2 2 2 2 2 272/300 Read block 3946 len 12288 273/300 Write block 4911 len 16384 data: 2 2 2 2 274/300 Read block 1590 len 8192 275/300 Write block 366 len 8192 data: 2 2 276/300 Read block 361 len 8192 277/300 Write block 1132 len 28672 data: 2 2 2 2 2 2 2 278/300 Read block 3227 len 28672 279/300 Read block 3337 len 12288 280/300 Read block 3736 len 12288 281/300 Write block 2888 len 4096 data: 2 282/300 Write block 2479 len 20480 data: 2 2 2 2 2 283/300 Write block 1011 len 28672 data: 3 3 3 3 2 2 2 284/300 Read block 268 len 4096 285/300 Write block 4643 len 12288 data: 2 2 2 286/300 Read block 4973 len 32768 287/300 Read block 908 len 20480 288/300 Read block 3060 len 28672 289/300 Read block 692 len 28672 290/300 Write block 1276 len 40960 data: 2 2 2 2 2 2 2 3 3 2 291/300 Read block 2543 len 4096 292/300 Flush 293/300 Read block 1867 len 16384 294/300 Write block 4515 len 32768 data: 2 2 2 2 2 2 2 2 295/300 Write block 725 len 32768 data: 2 2 2 2 2 2 2 2 296/300 Write block 2816 len 4096 data: 2 297/300 Read block 4298 len 36864 298/300 Read block 2312 len 20480 299/300 Write block 4143 len 36864 data: 2 2 2 2 2 2 2 2 2 300/300 Write block 3259 len 40960 data: 2 2 2 2 2 2 2 2 2 2 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:30:31.151 INFO request completed, latency_us: 369, response_code: 204, uri: /start/cid/2, method: POST, req_id: c4cd2109-e29a-4a47-b97f-13e2641790f0, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Write 9 Done Done New true 2 Acked 1001 Read 4 Done Done New true 3 Acked 1002 Read 1 Done Done New false 4 Acked 1003 Read 2 Done Done New false 5 Acked 1004 Read 1 Done Done New false 6 Acked 1005 Write 5 Done Done New false 7 Acked 1006 Read 1 Done Done New false 8 Acked 1007 Read 7 Done Done New false 9 Acked 1008 Write 10 Done Done New false 10 Acked 1009 Write 9 Done Done New false 11 Acked 1010 Write 2 Done Done New false 12 Acked 1011 Flush 0 Done Done New false 13 Acked 1012 Write 3 Done Done New false 14 Acked 1013 Read 9 Done Done New false 15 Acked 1014 Read 8 Done Done New false 16 Acked 1015 Write 9 Done Done New false 17 Acked 1016 Flush 0 Done Done New false 18 Acked 1017 Write 4 Done Done New false 19 Acked 1018 Write 3 Done Done New false 20 Acked 1019 Read 1 Done Done New false 21 Acked 1020 Read 9 Done Done New false 22 Acked 1021 Read 4 Done Done New false 23 Acked 1022 Write 3 Done Done New false 24 Acked 1023 Flush 0 Done Done New false 25 Acked 1024 Read 4 Done Done New false 26 Acked 1025 Write 3 Done Done New false 27 Acked 1026 Read 6 Done Done New false 28 Acked 1027 Flush 0 Done Done New false 29 Acked 1028 Write 7 Done Done New false 30 Acked 1029 Read 10 Done Done New false 31 Acked 1030 Read 5 Done Done New false 32 Acked 1031 Read 4 Done Done New false 33 Acked 1032 Read 9 Done Done New false 34 Acked 1033 Flush 0 Done Done New false 35 Acked 1034 Read 8 Done Done New false 36 Acked 1035 Write 7 Done Done New false 37 Acked 1036 Write 2 Done Done New false 38 Acked 1037 Write 6 Done Done New false 39 Acked 1038 Read 6 Done Done New false 40 Acked 1039 Read 8 Done Done New false 41 Acked 1040 Write 1 Done Done New false 42 Acked 1041 Read 8 Done Done New false 43 Acked 1042 Write 2 Done Done New false 44 Acked 1043 Read 8 Done Done New false 45 Acked 1044 Write 10 Done Done New false 46 Acked 1045 Read 4 Done Done New false 47 Acked 1046 Write 3 Done Done New false 48 Acked 1047 Read 7 Done Done New false 49 Acked 1048 Write 4 Done Done New false 50 Acked 1049 Read 3 Done Done New false 51 Acked 1050 Read 10 Done Done New false 52 Acked 1051 Read 7 Done Done New false 53 Acked 1052 Write 5 Done Done New false 54 Acked 1053 Write 6 Done Done New false 55 Acked 1054 Write 2 Done Done New false 56 Acked 1055 Read 6 Done Done New false 57 Acked 1056 Read 10 Done Done New false 58 Acked 1057 Read 1 Done Done New false 59 Acked 1058 Write 3 Done Done New false 60 Acked 1059 Write 7 Done Done New false 61 Acked 1060 Read 9 Done Done New false 62 Acked 1061 Write 3 Done Done New false 63 Acked 1062 Write 10 Done Done New false 64 Acked 1063 Write 3 Done Done New false 65 Acked 1064 Read 3 Done Done New false 66 Acked 1065 Write 5 Done Done New false 67 Acked 1066 Read 7 Done Done New false 68 Acked 1067 Write 6 Done Done New false 69 Acked 1068 Flush 0 Done Done New false 70 Acked 1069 Read 6 Done Done New false 71 Acked 1070 Read 2 Done Done New false 72 Acked 1071 Read 4 Done Done New false 73 Acked 1072 Read 7 Done Done New false 74 Acked 1073 Write 2 Done Done New false 75 Acked 1074 Read 10 Done Done New false 76 Acked 1075 Write 6 Done Done New false 77 Acked 1076 Read 10 Done Done New false 78 Acked 1077 Read 2 Done Done New false 79 Acked 1078 Read 2 Done Done New false 80 Acked 1079 Write 9 Done Done New false 81 Acked 1080 Read 7 Done Done New false 82 Acked 1081 Write 8 Done Done New false 83 Acked 1082 Write 4 Done Done New false 84 Acked 1083 Write 6 Done Done New false 85 Acked 1084 Read 5 Done Done New false 86 Acked 1085 Read 3 Done Done New false 87 Acked 1086 Write 2 Done Done New false 88 Acked 1087 Write 9 Done Done New false 89 Acked 1088 Write 10 Done Done New false 90 Acked 1089 Read 3 Done Done New false 91 Acked 1090 Write 1 Done Done New false 92 Acked 1091 Flush 0 Done Done New false 93 Acked 1092 Flush 0 Done Done New false 94 Acked 1093 Write 8 Done Done New false 95 Acked 1094 Read 7 Done Done New false 96 Acked 1095 Read 9 Done Done New false 97 Acked 1096 Write 1 Done Done New false 98 Acked 1097 Write 1 Done Done New false 99 Acked 1098 Read 10 Done Done New false 100 Acked 1099 Read 5 Done Done New false 101 Acked 1100 Write 5 Done Done New false 102 Acked 1101 Write 4 Done Done New false 103 Acked 1102 Write 6 Done Done New false 104 Acked 1103 Read 4 Done Done New false 105 Acked 1104 Read 4 Done Done New false 106 Acked 1105 Write 9 Done Done New false 107 Acked 1106 Read 8 Done Done New false 108 Acked 1107 Read 3 Done Done New false 109 Acked 1108 Write 10 Done Done New false 110 Acked 1109 Read 4 Done Done New false 111 Acked 1110 Write 8 Done Done New false 112 Acked 1111 Read 2 Done Done New false 113 Acked 1112 Flush 0 Done Done New false 114 Acked 1113 Read 2 Done Done New false 115 Acked 1114 Flush 0 Done Done New false 116 Acked 1115 Write 7 Done Done New false 117 Acked 1116 Read 3 Done Done New false 118 Acked 1117 Read 9 Done Done New false 119 Acked 1118 Write 4 Done Done New false 120 Acked 1119 Write 9 Done Done New false 121 Acked 1120 Write 8 Done Done New false 122 Acked 1121 Read 2 Done Done New false 123 Acked 1122 Read 7 Done Done New false 124 Acked 1123 Read 1 Done Done New false 125 Acked 1124 Read 9 Done Done New false 126 Acked 1125 Write 3 Done Done New false 127 Acked 1126 Write 3 Done Done New false 128 Acked 1127 Write 5 Done Done New false 129 Acked 1128 Write 9 Done Done New false 130 Acked 1129 Write 7 Done Done New false 131 Acked 1130 Write 1 Done Done New false 132 Acked 1131 Read 6 Done Done New false 133 Acked 1132 Write 7 Done Done New false 134 Acked 1133 Read 6 Done Done New false 135 Acked 1134 Read 6 Done Done New false 136 Acked 1135 Flush 0 Done Done New false 137 Acked 1136 Read 5 Done Done New false 138 Acked 1137 Read 2 Done Done New false 139 Acked 1138 Write 4 Done Done New false 140 Acked 1139 Read 4 Done Done New false 141 Acked 1140 Read 4 Done Done New false 142 Acked 1141 Write 1 Done Done New false 143 Acked 1142 Flush 0 Done Done New false 144 Acked 1143 Read 6 Done Done New false 145 Acked 1144 Read 8 Done Done New false 146 Acked 1145 Write 3 Done Done New false 147 Acked 1146 Flush 0 Done Done New false 148 Acked 1147 Read 3 Done Done New false 149 Acked 1148 Read 1 Done Done New false 150 Acked 1149 Read 5 Done Done New false 151 Acked 1150 Read 1 Done Done New false 152 Acked 1151 Read 10 Done Done New false 153 Acked 1152 Read 7 Done Done New false 154 Acked 1153 Write 8 Done Done New false 155 Acked 1154 Read 10 Done Done New false 156 Acked 1155 Write 8 Done Done New false 157 Acked 1156 Flush 0 Done Done New false 158 Acked 1157 Write 2 Done Done New false 159 Acked 1158 Read 9 Done Done New false 160 Acked 1159 Write 10 Done Done New false 161 Acked 1160 Read 2 Done Done New false 162 Acked 1161 Write 3 Done Done New false 163 Acked 1162 Flush 0 Done Done New false 164 Acked 1163 Read 4 Done Done New false 165 Acked 1164 Flush 0 Done Done New false 166 Acked 1165 Read 3 Done Done New false 167 Acked 1166 Read 10 Done Done New false 168 Acked 1167 Read 5 Done Done New false 169 Acked 1168 Read 8 Done Done New false 170 Acked 1169 Write 10 Done Done New false 171 Acked 1170 Write 2 Done Done New false 172 Acked 1171 Read 10 Done Done New false 173 Acked 1172 Flush 0 Done Done New false 174 Acked 1173 Read 8 Done Done New false 175 Acked 1174 Read 5 Done Done New false 176 Acked 1175 Read 8 Done Done New false 177 Acked 1176 Read 5 Done Done New false 178 Acked 1177 Read 6 Done Done New false 179 Acked 1178 Read 1 Done Done New false 180 Acked 1179 Read 9 Done Done New false 181 Acked 1180 Read 6 Done Done New false 182 Acked 1181 Read 1 Done Done New false 183 Acked 1182 Flush 0 Done Done New false 184 Acked 1183 Flush 0 Done Done New false 185 Acked 1184 Write 8 Done Done New false 186 Acked 1185 Read 1 Done Done New false 187 Acked 1186 Write 1 Done Done New false 188 Acked 1187 Read 5 Done Done New false 189 Acked 1188 Read 8 Done Done New false 190 Acked 1189 Flush 0 Done Done New false 191 Acked 1190 Write 5 Done Done New false 192 Acked 1191 Read 5 Done Done New false 193 Acked 1192 Flush 0 Done Done New false 194 Acked 1193 Read 4 Done Done New false 195 Acked 1194 Flush 0 Done Done New false 196 Acked 1195 Flush 0 Done Done New false 197 Acked 1196 Read 1 Done Done New false 198 Acked 1197 Read 6 Done Done New false 199 Acked 1198 Write 10 Done Done New false 200 Acked 1199 Write 9 Done Done New false 201 Acked 1200 Read 9 Done Done New false 202 Acked 1201 Read 8 Done Done New false 203 Acked 1202 Read 4 Done Done New false 204 Acked 1203 Read 1 Done Done New false 205 Acked 1204 Write 6 Done Done New false 206 Acked 1205 Flush 0 Done Done New false 207 Acked 1206 Read 10 Done Done New false 208 Acked 1207 Write 7 Done Done New false 209 Acked 1208 Write 4 Done Done New false 210 Acked 1209 Read 8 Done Done New false 211 Acked 1210 Read 10 Done Done New false 212 Acked 1211 Read 8 Done Done New false 213 Acked 1212 Read 9 Done Done New false 214 Acked 1213 Write 6 Done Done New false 215 Acked 1214 Read 5 Done Done New false 216 Acked 1215 Write 8 Done Done New false 217 Acked 1216 Read 7 Done Done New false 218 Acked 1217 Read 3 Done Done New false 219 Acked 1218 Read 9 Done Done New false 220 Acked 1219 Write 1 Done Done New false 221 Acked 1220 Write 10 Done Done New false 222 Acked 1221 Write 6 Done Done New false 223 Acked 1222 Read 6 Done Done New false 224 Acked 1223 Read 2 Done Done New false 225 Acked 1224 Write 4 Done Done New false 226 Acked 1225 Write 10 Done Done New false 227 Acked 1226 Write 5 Done Done New false 228 Acked 1227 Read 7 Done Done New false 229 Acked 1228 Read 5 Done Done New false 230 Acked 1229 Read 4 Done Done New false 231 Acked 1230 Write 2 Done Done New false 232 Acked 1231 Write 6 Done Done New false 233 Acked 1232 Read 1 Done Done New false 234 Acked 1233 Read 5 Done Done New false 235 Acked 1234 Write 8 Done Done New false 236 Acked 1235 Write 4 Done Done New false 237 Acked 1236 Read 5 Done Done New false 238 Acked 1237 Flush 0 Done Done New false 239 Acked 1238 Read 5 Done Done New false 240 Acked 1239 Flush 0 Done Done New false 241 Acked 1240 Flush 0 Done Done New false 242 Acked 1241 Read 5 Done Done New false 243 Acked 1242 Read 7 Done Done New false 244 Acked 1243 Write 2 Done Done New false 245 Acked 1244 Write 1 Done Done New false 246 Acked 1245 Write 5 Done Done New false 247 Acked 1246 Read 6 Done Done New false 248 Acked 1247 Write 2 Done Done New false 249 Acked 1248 Read 2 Done Done New false 250 Acked 1249 Write 9 Done Done New false 251 Acked 1250 Write 1 Done Done New false 252 Acked 1251 Write 10 Done Done New false 253 Acked 1252 Write 6 Done Done New false 254 Acked 1253 Write 2 Done Done New false 255 Acked 1254 Write 3 Done Done New false 256 Acked 1255 Read 9 Done Done New false 257 Acked 1256 Read 5 Done Done New false 258 Acked 1257 Read 2 Done Done New false 259 Acked 1258 Flush 0 Done Done New false 260 Acked 1259 Write 4 Done Done New false 261 Acked 1260 Flush 0 Done Done New false 262 Acked 1261 Read 7 Done Done New false 263 Acked 1262 Read 2 Done Done New false 264 Acked 1263 Write 2 Done Done New false 265 Acked 1264 Read 3 Done Done New false 266 Acked 1265 Flush 0 Done Done New false 267 Acked 1266 Read 6 Done Done New false 268 Acked 1267 Read 5 Done Done New false 269 Acked 1268 Write 5 Done Done New false 270 Acked 1269 Write 4 Done Done New false 271 Acked 1270 Write 5 Done Done New false 272 Acked 1271 Read 3 Done Done New false 273 Acked 1272 Write 4 Done Done New false 274 Acked 1273 Read 2 Done Done New false 275 Acked 1274 Write 2 Done Done New false 276 Acked 1275 Read 2 Done Done New false 277 Acked 1276 Write 7 Done Done New false 278 Acked 1277 Read 7 Done Done New false 279 Acked 1278 Read 3 Done Done New false 280 Acked 1279 Read 3 Done Done New false 281 Acked 1280 Write 1 Done Done New false 282 Acked 1281 Write 5 Done Done New false 283 Acked 1282 Write 7 Done Done New false 284 Acked 1283 Read 1 Done Done New false 285 Acked 1284 Write 3 Done Done New false 286 Acked 1285 Read 8 Done Done New false 287 Acked 1286 Read 5 Done Done New false 288 Acked 1287 Read 7 Done Done New false 289 Acked 1288 Read 7 Done Done New false 290 Acked 1289 Write 10 Done Done New false 291 Acked 1290 Read 1 Done Done New false 292 Acked 1291 Flush 0 Done Done New false 293 Acked 1292 Read 4 Done Done New false 294 Acked 1293 Write 8 Done Done New false 295 Acked 1294 Write 8 Done Done New false 296 Acked 1295 Write 1 Done Done New false 297 Acked 1296 Read 9 Done Done New false 298 Acked 1297 Read 5 Done Done New false 299 Acked 1298 Write 9 Done Done New false 300 Acked 1299 Write 10 Done Sent New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 300 300 Sent 0 1 0 1 Done 300 299 0 599 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1291 1291 0 Downstairs last five completed: Upstairs last five completed: 300 299 298 297 296 CLIENT: Up:0 ds:300 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:31.266638595Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:31.266752019Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:31.266813893Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:31.266838204Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:31.266864277Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:31.266884553Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:31.266906878Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:31.267108702Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:32.267989684Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:32.268060303Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:32.268088517Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:32.268122974Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:32.268144506Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:32.268209308Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:32.268441856Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:33.270375656Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:33.270446972Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:33.270475033Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:33.270501557Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:33.270521922Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:33.270630826Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:33.27085569Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:34.271768721Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:34.271882225Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:34.271943084Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:34.27202928Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:34.272051336Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:34.272074003Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:34.272260493Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Write 9 Done Done New true 2 Acked 1001 Read 4 Done Done New true 3 Acked 1002 Read 1 Done Done New true 4 Acked 1003 Read 2 Done Done New true 5 Acked 1004 Read 1 Done Done New true 6 Acked 1005 Write 5 Done Done New true 7 Acked 1006 Read 1 Done Done New true 8 Acked 1007 Read 7 Done Done New true 9 Acked 1008 Write 10 Done Done New true 10 Acked 1009 Write 9 Done Done New true 11 Acked 1010 Write 2 Done Done New true 12 Acked 1011 Flush 0 Done Done New true 13 Acked 1012 Write 3 Done Done New true 14 Acked 1013 Read 9 Done Done New true 15 Acked 1014 Read 8 Done Done New true 16 Acked 1015 Write 9 Done Done New true 17 Acked 1016 Flush 0 Done Done New true 18 Acked 1017 Write 4 Done Done New true 19 Acked 1018 Write 3 Done Done New true 20 Acked 1019 Read 1 Done Done New true 21 Acked 1020 Read 9 Done Done New true 22 Acked 1021 Read 4 Done Done New true 23 Acked 1022 Write 3 Done Done New true 24 Acked 1023 Flush 0 Done Done New true 25 Acked 1024 Read 4 Done Done New true 26 Acked 1025 Write 3 Done Done New true 27 Acked 1026 Read 6 Done Done New true 28 Acked 1027 Flush 0 Done Done New true 29 Acked 1028 Write 7 Done Done New true 30 Acked 1029 Read 10 Done Done New true 31 Acked 1030 Read 5 Done Done New true 32 Acked 1031 Read 4 Done Done New true 33 Acked 1032 Read 9 Done Done New true 34 Acked 1033 Flush 0 Done Done New true 35 Acked 1034 Read 8 Done Done New true 36 Acked 1035 Write 7 Done Done New true 37 Acked 1036 Write 2 Done Done New true 38 Acked 1037 Write 6 Done Done New true 39 Acked 1038 Read 6 Done Done New true 40 Acked 1039 Read 8 Done Done New true 41 Acked 1040 Write 1 Done Done New true 42 Acked 1041 Read 8 Done Done New true 43 Acked 1042 Write 2 Done Done New true 44 Acked 1043 Read 8 Done Done New true 45 Acked 1044 Write 10 Done Done New true 46 Acked 1045 Read 4 Done Done New true 47 Acked 1046 Write 3 Done Done New true 48 Acked 1047 Read 7 Done Done New true 49 Acked 1048 Write 4 Done Done New true 50 Acked 1049 Read 3 Done Done New true 51 Acked 1050 Read 10 Done Done New true 52 Acked 1051 Read 7 Done Done New true 53 Acked 1052 Write 5 Done Done New true 54 Acked 1053 Write 6 Done Done New true 55 Acked 1054 Write 2 Done Done New true 56 Acked 1055 Read 6 Done Done New true 57 Acked 1056 Read 10 Done Done New true 58 Acked 1057 Read 1 Done Done New true 59 Acked 1058 Write 3 Done Done New true 60 Acked 1059 Write 7 Done Done New true 61 Acked 1060 Read 9 Done Done New true 62 Acked 1061 Write 3 Done Done New true 63 Acked 1062 Write 10 Done Done New true 64 Acked 1063 Write 3 Done Done New true 65 Acked 1064 Read 3 Done Done New true 66 Acked 1065 Write 5 Done Done New true 67 Acked 1066 Read 7 Done Done New true 68 Acked 1067 Write 6 Done Done New true 69 Acked 1068 Flush 0 Done Done New true 70 Acked 1069 Read 6 Done Done New true 71 Acked 1070 Read 2 Done Done New true 72 Acked 1071 Read 4 Done Done New true 73 Acked 1072 Read 7 Done Done New true 74 Acked 1073 Write 2 Done Done New true 75 Acked 1074 Read 10 Done Done New true 76 Acked 1075 Write 6 Done Done New true 77 Acked 1076 Read 10 Done Done New true 78 Acked 1077 Read 2 Done Done New true 79 Acked 1078 Read 2 Done Done New true 80 Acked 1079 Write 9 Done Done New true 81 Acked 1080 Read 7 Done Done New true 82 Acked 1081 Write 8 Done Done New true 83 Acked 1082 Write 4 Done Done New true 84 Acked 1083 Write 6 Done Done New true 85 Acked 1084 Read 5 Done Done New true 86 Acked 1085 Read 3 Done Done New true 87 Acked 1086 Write 2 Done Done New true 88 Acked 1087 Write 9 Done Done New true 89 Acked 1088 Write 10 Done Done New true 90 Acked 1089 Read 3 Done Done New true 91 Acked 1090 Write 1 Done Done New true 92 Acked 1091 Flush 0 Done Done New true 93 Acked 1092 Flush 0 Done Done New true 94 Acked 1093 Write 8 Done Done New true 95 Acked 1094 Read 7 Done Done New true 96 Acked 1095 Read 9 Done Done New true 97 Acked 1096 Write 1 Done Done New true 98 Acked 1097 Write 1 Done Done New true 99 Acked 1098 Read 10 Done Done New true 100 Acked 1099 Read 5 Done Done New true 101 Acked 1100 Write 5 Done Done New true 102 Acked 1101 Write 4 Done Done New true 103 Acked 1102 Write 6 Done Done New true 104 Acked 1103 Read 4 Done Done New true 105 Acked 1104 Read 4 Done Done New true 106 Acked 1105 Write 9 Done Done New true 107 Acked 1106 Read 8 Done Done New true 108 Acked 1107 Read 3 Done Done New true 109 Acked 1108 Write 10 Done Done New true 110 Acked 1109 Read 4 Done Done New true 111 Acked 1110 Write 8 Done Done New true 112 Acked 1111 Read 2 Done Done New true 113 Acked 1112 Flush 0 Done Done New true 114 Acked 1113 Read 2 Done Done New true 115 Acked 1114 Flush 0 Done Done New true 116 Acked 1115 Write 7 Done Done New true 117 Acked 1116 Read 3 Done Done New true 118 Acked 1117 Read 9 Done Done New true 119 Acked 1118 Write 4 Done Done New true 120 Acked 1119 Write 9 Done Done New true 121 Acked 1120 Write 8 Done Done New true 122 Acked 1121 Read 2 Done Done New true 123 Acked 1122 Read 7 Done Done New true 124 Acked 1123 Read 1 Done Done New true 125 Acked 1124 Read 9 Done Done New true 126 Acked 1125 Write 3 Done Done New true 127 Acked 1126 Write 3 Done Done New true 128 Acked 1127 Write 5 Done Done New true 129 Acked 1128 Write 9 Done Done New true 130 Acked 1129 Write 7 Done Done New true 131 Acked 1130 Write 1 Done Done New true 132 Acked 1131 Read 6 Done Done New true 133 Acked 1132 Write 7 Done Done New true 134 Acked 1133 Read 6 Done Done New true 135 Acked 1134 Read 6 Done Done New true 136 Acked 1135 Flush 0 Done Done New true 137 Acked 1136 Read 5 Done Done New true 138 Acked 1137 Read 2 Done Done New true 139 Acked 1138 Write 4 Done Done New true 140 Acked 1139 Read 4 Done Done New true 141 Acked 1140 Read 4 Done Done New true 142 Acked 1141 Write 1 Done Done New true 143 Acked 1142 Flush 0 Done Done New true 144 Acked 1143 Read 6 Done Done New true 145 Acked 1144 Read 8 Done Done New true 146 Acked 1145 Write 3 Done Done New true 147 Acked 1146 Flush 0 Done Done New true 148 Acked 1147 Read 3 Done Done New true 149 Acked 1148 Read 1 Done Done New true 150 Acked 1149 Read 5 Done Done New true 151 Acked 1150 Read 1 Done Done New true 152 Acked 1151 Read 10 Done Done New true 153 Acked 1152 Read 7 Done Done New true 154 Acked 1153 Write 8 Done Done New true 155 Acked 1154 Read 10 Done Done New true 156 Acked 1155 Write 8 Done Done New true 157 Acked 1156 Flush 0 Done Done New true 158 Acked 1157 Write 2 Done Done New true 159 Acked 1158 Read 9 Done Done New true 160 Acked 1159 Write 10 Done Done New true 161 Acked 1160 Read 2 Done Done New true 162 Acked 1161 Write 3 Done Done New true 163 Acked 1162 Flush 0 Done Done New true 164 Acked 1163 Read 4 Done Done New true 165 Acked 1164 Flush 0 Done Done New true 166 Acked 1165 Read 3 Done Done New true 167 Acked 1166 Read 10 Done Done New true 168 Acked 1167 Read 5 Done Done New true 169 Acked 1168 Read 8 Done Done New true 170 Acked 1169 Write 10 Done Done New true 171 Acked 1170 Write 2 Done Done New true 172 Acked 1171 Read 10 Done Done New true 173 Acked 1172 Flush 0 Done Done New true 174 Acked 1173 Read 8 Done Done New true 175 Acked 1174 Read 5 Done Done New true 176 Acked 1175 Read 8 Done Done New true 177 Acked 1176 Read 5 Done Done New true 178 Acked 1177 Read 6 Done Done New true 179 Acked 1178 Read 1 Done Done New true 180 Acked 1179 Read 9 Done Done New true 181 Acked 1180 Read 6 Done Done New true 182 Acked 1181 Read 1 Done Done New true 183 Acked 1182 Flush 0 Done Done New true 184 Acked 1183 Flush 0 Done Done New true 185 Acked 1184 Write 8 Done Done New true 186 Acked 1185 Read 1 Done Done New true 187 Acked 1186 Write 1 Done Done New true 188 Acked 1187 Read 5 Done Done New true 189 Acked 1188 Read 8 Done Done New true 190 Acked 1189 Flush 0 Done Done New true 191 Acked 1190 Write 5 Done Done New true 192 Acked 1191 Read 5 Done Done New true 193 Acked 1192 Flush 0 Done Done New true 194 Acked 1193 Read 4 Done Done New true 195 Acked 1194 Flush 0 Done Done New true 196 Acked 1195 Flush 0 Done Done New true 197 Acked 1196 Read 1 Done Done New true 198 Acked 1197 Read 6 Done Done New true 199 Acked 1198 Write 10 Done Done New true 200 Acked 1199 Write 9 Done Done New true 201 Acked 1200 Read 9 Done Done New true 202 Acked 1201 Read 8 Done Done New true 203 Acked 1202 Read 4 Done Done New true 204 Acked 1203 Read 1 Done Done New true 205 Acked 1204 Write 6 Done Done New true 206 Acked 1205 Flush 0 Done Done New true 207 Acked 1206 Read 10 Done Done New true 208 Acked 1207 Write 7 Done Done New true 209 Acked 1208 Write 4 Done Done New true 210 Acked 1209 Read 8 Done Done New true 211 Acked 1210 Read 10 Done Done New true 212 Acked 1211 Read 8 Done Done New true 213 Acked 1212 Read 9 Done Done New true 214 Acked 1213 Write 6 Done Done New true 215 Acked 1214 Read 5 Done Done New true 216 Acked 1215 Write 8 Done Done New true 217 Acked 1216 Read 7 Done Done New true 218 Acked 1217 Read 3 Done Done New true 219 Acked 1218 Read 9 Done Done New true 220 Acked 1219 Write 1 Done Done New true 221 Acked 1220 Write 10 Done Done New true 222 Acked 1221 Write 6 Done Done New true 223 Acked 1222 Read 6 Done Done New true 224 Acked 1223 Read 2 Done Done New true 225 Acked 1224 Write 4 Done Done New true 226 Acked 1225 Write 10 Done Done New true 227 Acked 1226 Write 5 Done Done New true 228 Acked 1227 Read 7 Done Done New true 229 Acked 1228 Read 5 Done Done New true 230 Acked 1229 Read 4 Done Done New true 231 Acked 1230 Write 2 Done Done New true 232 Acked 1231 Write 6 Done Done New true 233 Acked 1232 Read 1 Done Done New true 234 Acked 1233 Read 5 Done Done New true 235 Acked 1234 Write 8 Done Done New true 236 Acked 1235 Write 4 Done Done New true 237 Acked 1236 Read 5 Done Done New true 238 Acked 1237 Flush 0 Done Done New true 239 Acked 1238 Read 5 Done Done New true 240 Acked 1239 Flush 0 Done Done New true 241 Acked 1240 Flush 0 Done Done New true 242 Acked 1241 Read 5 Done Done New true 243 Acked 1242 Read 7 Done Done New true 244 Acked 1243 Write 2 Done Done New true 245 Acked 1244 Write 1 Done Done New true 246 Acked 1245 Write 5 Done Done New true 247 Acked 1246 Read 6 Done Done New true 248 Acked 1247 Write 2 Done Done New true 249 Acked 1248 Read 2 Done Done New true 250 Acked 1249 Write 9 Done Done New true 251 Acked 1250 Write 1 Done Done New true 252 Acked 1251 Write 10 Done Done New true 253 Acked 1252 Write 6 Done Done New true 254 Acked 1253 Write 2 Done Done New true 255 Acked 1254 Write 3 Done Done New true 256 Acked 1255 Read 9 Done Done New true 257 Acked 1256 Read 5 Done Done New true 258 Acked 1257 Read 2 Done Done New true 259 Acked 1258 Flush 0 Done Done New true 260 Acked 1259 Write 4 Done Done New true 261 Acked 1260 Flush 0 Done Done New true 262 Acked 1261 Read 7 Done Done New true 263 Acked 1262 Read 2 Done Done New true 264 Acked 1263 Write 2 Done Done New true 265 Acked 1264 Read 3 Done Done New true 266 Acked 1265 Flush 0 Done Done New true 267 Acked 1266 Read 6 Done Done New true 268 Acked 1267 Read 5 Done Done New true 269 Acked 1268 Write 5 Done Done New true 270 Acked 1269 Write 4 Done Done New true 271 Acked 1270 Write 5 Done Done New true 272 Acked 1271 Read 3 Done Done New true 273 Acked 1272 Write 4 Done Done New true 274 Acked 1273 Read 2 Done Done New true 275 Acked 1274 Write 2 Done Done New true 276 Acked 1275 Read 2 Done Done New true 277 Acked 1276 Write 7 Done Done New true 278 Acked 1277 Read 7 Done Done New true 279 Acked 1278 Read 3 Done Done New true 280 Acked 1279 Read 3 Done Done New true 281 Acked 1280 Write 1 Done Done New true 282 Acked 1281 Write 5 Done Done New true 283 Acked 1282 Write 7 Done Done New true 284 Acked 1283 Read 1 Done Done New true 285 Acked 1284 Write 3 Done Done New true 286 Acked 1285 Read 8 Done Done New true 287 Acked 1286 Read 5 Done Done New true 288 Acked 1287 Read 7 Done Done New true 289 Acked 1288 Read 7 Done Done New true 290 Acked 1289 Write 10 Done Done New true 291 Acked 1290 Read 1 Done Done New true 292 Acked 1291 Flush 0 Done Done New true 293 Acked 1292 Read 4 Done Done New true 294 Acked 1293 Write 8 Done Done New true 295 Acked 1294 Write 8 Done Done New true 296 Acked 1295 Write 1 Done Done New true 297 Acked 1296 Read 9 Done Done New true 298 Acked 1297 Read 5 Done Done New true 299 Acked 1298 Write 9 Done Done New true 300 Acked 1299 Write 10 Done Done New true 301 Acked 1300 Flush 0 Done Done New true STATES DS:0 DS:1 DS:2 TOTAL New 0 0 301 301 Sent 0 0 0 0 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1300 1300 0 Downstairs last five completed: Upstairs last five completed: 301 300 299 298 297 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:35.272562317Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:35.272626403Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:35.272654311Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:35.272680756Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:35.272700963Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:35.272723259Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:35.272929938Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:36.273624376Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:36.273693662Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:36.273722255Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:36.273746741Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:36.273773111Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:36.273793595Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:36.273826271Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:36.274040413Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:37.274977029Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:37.275086273Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:37.275146999Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:37.275205531Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:37.275244388Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:37.275287558Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:37.2755495Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186) [8830][2] reports Running {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:38.277335061Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:38.27740345Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:38.277431788Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:38.277458146Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:38.277486127Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:38.277509764Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:38.277735174Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Write 9 Done Done New true 2 Acked 1001 Read 4 Done Done New true 3 Acked 1002 Read 1 Done Done New true 4 Acked 1003 Read 2 Done Done New true 5 Acked 1004 Read 1 Done Done New true 6 Acked 1005 Write 5 Done Done New true 7 Acked 1006 Read 1 Done Done New true 8 Acked 1007 Read 7 Done Done New true 9 Acked 1008 Write 10 Done Done New true 10 Acked 1009 Write 9 Done Done New true 11 Acked 1010 Write 2 Done Done New true 12 Acked 1011 Flush 0 Done Done New true 13 Acked 1012 Write 3 Done Done New true 14 Acked 1013 Read 9 Done Done New true 15 Acked 1014 Read 8 Done Done New true 16 Acked 1015 Write 9 Done Done New true 17 Acked 1016 Flush 0 Done Done New true 18 Acked 1017 Write 4 Done Done New true 19 Acked 1018 Write 3 Done Done New true 20 Acked 1019 Read 1 Done Done New true 21 Acked 1020 Read 9 Done Done New true 22 Acked 1021 Read 4 Done Done New true 23 Acked 1022 Write 3 Done Done New true 24 Acked 1023 Flush 0 Done Done New true 25 Acked 1024 Read 4 Done Done New true 26 Acked 1025 Write 3 Done Done New true 27 Acked 1026 Read 6 Done Done New true 28 Acked 1027 Flush 0 Done Done New true 29 Acked 1028 Write 7 Done Done New true 30 Acked 1029 Read 10 Done Done New true 31 Acked 1030 Read 5 Done Done New true 32 Acked 1031 Read 4 Done Done New true 33 Acked 1032 Read 9 Done Done New true 34 Acked 1033 Flush 0 Done Done New true 35 Acked 1034 Read 8 Done Done New true 36 Acked 1035 Write 7 Done Done New true 37 Acked 1036 Write 2 Done Done New true 38 Acked 1037 Write 6 Done Done New true 39 Acked 1038 Read 6 Done Done New true 40 Acked 1039 Read 8 Done Done New true 41 Acked 1040 Write 1 Done Done New true 42 Acked 1041 Read 8 Done Done New true 43 Acked 1042 Write 2 Done Done New true 44 Acked 1043 Read 8 Done Done New true 45 Acked 1044 Write 10 Done Done New true 46 Acked 1045 Read 4 Done Done New true 47 Acked 1046 Write 3 Done Done New true 48 Acked 1047 Read 7 Done Done New true 49 Acked 1048 Write 4 Done Done New true 50 Acked 1049 Read 3 Done Done New true 51 Acked 1050 Read 10 Done Done New true 52 Acked 1051 Read 7 Done Done New true 53 Acked 1052 Write 5 Done Done New true 54 Acked 1053 Write 6 Done Done New true 55 Acked 1054 Write 2 Done Done New true 56 Acked 1055 Read 6 Done Done New true 57 Acked 1056 Read 10 Done Done New true 58 Acked 1057 Read 1 Done Done New true 59 Acked 1058 Write 3 Done Done New true 60 Acked 1059 Write 7 Done Done New true 61 Acked 1060 Read 9 Done Done New true 62 Acked 1061 Write 3 Done Done New true 63 Acked 1062 Write 10 Done Done New true 64 Acked 1063 Write 3 Done Done New true 65 Acked 1064 Read 3 Done Done New true 66 Acked 1065 Write 5 Done Done New true 67 Acked 1066 Read 7 Done Done New true 68 Acked 1067 Write 6 Done Done New true 69 Acked 1068 Flush 0 Done Done New true 70 Acked 1069 Read 6 Done Done New true 71 Acked 1070 Read 2 Done Done New true 72 Acked 1071 Read 4 Done Done New true 73 Acked 1072 Read 7 Done Done New true 74 Acked 1073 Write 2 Done Done New true 75 Acked 1074 Read 10 Done Done New true 76 Acked 1075 Write 6 Done Done New true 77 Acked 1076 Read 10 Done Done New true 78 Acked 1077 Read 2 Done Done New true 79 Acked 1078 Read 2 Done Done New true 80 Acked 1079 Write 9 Done Done New true 81 Acked 1080 Read 7 Done Done New true 82 Acked 1081 Write 8 Done Done New true 83 Acked 1082 Write 4 Done Done New true 84 Acked 1083 Write 6 Done Done New true 85 Acked 1084 Read 5 Done Done New true 86 Acked 1085 Read 3 Done Done New true 87 Acked 1086 Write 2 Done Done New true 88 Acked 1087 Write 9 Done Done New true 89 Acked 1088 Write 10 Done Done New true 90 Acked 1089 Read 3 Done Done New true 91 Acked 1090 Write 1 Done Done New true 92 Acked 1091 Flush 0 Done Done New true 93 Acked 1092 Flush 0 Done Done New true 94 Acked 1093 Write 8 Done Done New true 95 Acked 1094 Read 7 Done Done New true 96 Acked 1095 Read 9 Done Done New true 97 Acked 1096 Write 1 Done Done New true 98 Acked 1097 Write 1 Done Done New true 99 Acked 1098 Read 10 Done Done New true 100 Acked 1099 Read 5 Done Done New true 101 Acked 1100 Write 5 Done Done New true 102 Acked 1101 Write 4 Done Done New true 103 Acked 1102 Write 6 Done Done New true 104 Acked 1103 Read 4 Done Done New true 105 Acked 1104 Read 4 Done Done New true 106 Acked 1105 Write 9 Done Done New true 107 Acked 1106 Read 8 Done Done New true 108 Acked 1107 Read 3 Done Done New true 109 Acked 1108 Write 10 Done Done New true 110 Acked 1109 Read 4 Done Done New true 111 Acked 1110 Write 8 Done Done New true 112 Acked 1111 Read 2 Done Done New true 113 Acked 1112 Flush 0 Done Done New true 114 Acked 1113 Read 2 Done Done New true 115 Acked 1114 Flush 0 Done Done New true 116 Acked 1115 Write 7 Done Done New true 117 Acked 1116 Read 3 Done Done New true 118 Acked 1117 Read 9 Done Done New true 119 Acked 1118 Write 4 Done Done New true 120 Acked 1119 Write 9 Done Done New true 121 Acked 1120 Write 8 Done Done New true 122 Acked 1121 Read 2 Done Done New true 123 Acked 1122 Read 7 Done Done New true 124 Acked 1123 Read 1 Done Done New true 125 Acked 1124 Read 9 Done Done New true 126 Acked 1125 Write 3 Done Done New true 127 Acked 1126 Write 3 Done Done New true 128 Acked 1127 Write 5 Done Done New true 129 Acked 1128 Write 9 Done Done New true 130 Acked 1129 Write 7 Done Done New true 131 Acked 1130 Write 1 Done Done New true 132 Acked 1131 Read 6 Done Done New true 133 Acked 1132 Write 7 Done Done New true 134 Acked 1133 Read 6 Done Done New true 135 Acked 1134 Read 6 Done Done New true 136 Acked 1135 Flush 0 Done Done New true 137 Acked 1136 Read 5 Done Done New true 138 Acked 1137 Read 2 Done Done New true 139 Acked 1138 Write 4 Done Done New true 140 Acked 1139 Read 4 Done Done New true 141 Acked 1140 Read 4 Done Done New true 142 Acked 1141 Write 1 Done Done New true 143 Acked 1142 Flush 0 Done Done New true 144 Acked 1143 Read 6 Done Done New true 145 Acked 1144 Read 8 Done Done New true 146 Acked 1145 Write 3 Done Done New true 147 Acked 1146 Flush 0 Done Done New true 148 Acked 1147 Read 3 Done Done New true 149 Acked 1148 Read 1 Done Done New true 150 Acked 1149 Read 5 Done Done New true 151 Acked 1150 Read 1 Done Done New true 152 Acked 1151 Read 10 Done Done New true 153 Acked 1152 Read 7 Done Done New true 154 Acked 1153 Write 8 Done Done New true 155 Acked 1154 Read 10 Done Done New true 156 Acked 1155 Write 8 Done Done New true 157 Acked 1156 Flush 0 Done Done New true 158 Acked 1157 Write 2 Done Done New true 159 Acked 1158 Read 9 Done Done New true 160 Acked 1159 Write 10 Done Done New true 161 Acked 1160 Read 2 Done Done New true 162 Acked 1161 Write 3 Done Done New true 163 Acked 1162 Flush 0 Done Done New true 164 Acked 1163 Read 4 Done Done New true 165 Acked 1164 Flush 0 Done Done New true 166 Acked 1165 Read 3 Done Done New true 167 Acked 1166 Read 10 Done Done New true 168 Acked 1167 Read 5 Done Done New true 169 Acked 1168 Read 8 Done Done New true 170 Acked 1169 Write 10 Done Done New true 171 Acked 1170 Write 2 Done Done New true 172 Acked 1171 Read 10 Done Done New true 173 Acked 1172 Flush 0 Done Done New true 174 Acked 1173 Read 8 Done Done New true 175 Acked 1174 Read 5 Done Done New true 176 Acked 1175 Read 8 Done Done New true 177 Acked 1176 Read 5 Done Done New true 178 Acked 1177 Read 6 Done Done New true 179 Acked 1178 Read 1 Done Done New true 180 Acked 1179 Read 9 Done Done New true 181 Acked 1180 Read 6 Done Done New true 182 Acked 1181 Read 1 Done Done New true 183 Acked 1182 Flush 0 Done Done New true 184 Acked 1183 Flush 0 Done Done New true 185 Acked 1184 Write 8 Done Done New true 186 Acked 1185 Read 1 Done Done New true 187 Acked 1186 Write 1 Done Done New true 188 Acked 1187 Read 5 Done Done New true 189 Acked 1188 Read 8 Done Done New true 190 Acked 1189 Flush 0 Done Done New true 191 Acked 1190 Write 5 Done Done New true 192 Acked 1191 Read 5 Done Done New true 193 Acked 1192 Flush 0 Done Done New true 194 Acked 1193 Read 4 Done Done New true 195 Acked 1194 Flush 0 Done Done New true 196 Acked 1195 Flush 0 Done Done New true 197 Acked 1196 Read 1 Done Done New true 198 Acked 1197 Read 6 Done Done New true 199 Acked 1198 Write 10 Done Done New true 200 Acked 1199 Write 9 Done Done New true 201 Acked 1200 Read 9 Done Done New true 202 Acked 1201 Read 8 Done Done New true 203 Acked 1202 Read 4 Done Done New true 204 Acked 1203 Read 1 Done Done New true 205 Acked 1204 Write 6 Done Done New true 206 Acked 1205 Flush 0 Done Done New true 207 Acked 1206 Read 10 Done Done New true 208 Acked 1207 Write 7 Done Done New true 209 Acked 1208 Write 4 Done Done New true 210 Acked 1209 Read 8 Done Done New true 211 Acked 1210 Read 10 Done Done New true 212 Acked 1211 Read 8 Done Done New true 213 Acked 1212 Read 9 Done Done New true 214 Acked 1213 Write 6 Done Done New true 215 Acked 1214 Read 5 Done Done New true 216 Acked 1215 Write 8 Done Done New true 217 Acked 1216 Read 7 Done Done New true 218 Acked 1217 Read 3 Done Done New true 219 Acked 1218 Read 9 Done Done New true 220 Acked 1219 Write 1 Done Done New true 221 Acked 1220 Write 10 Done Done New true 222 Acked 1221 Write 6 Done Done New true 223 Acked 1222 Read 6 Done Done New true 224 Acked 1223 Read 2 Done Done New true 225 Acked 1224 Write 4 Done Done New true 226 Acked 1225 Write 10 Done Done New true 227 Acked 1226 Write 5 Done Done New true 228 Acked 1227 Read 7 Done Done New true 229 Acked 1228 Read 5 Done Done New true 230 Acked 1229 Read 4 Done Done New true 231 Acked 1230 Write 2 Done Done New true 232 Acked 1231 Write 6 Done Done New true 233 Acked 1232 Read 1 Done Done New true 234 Acked 1233 Read 5 Done Done New true 235 Acked 1234 Write 8 Done Done New true 236 Acked 1235 Write 4 Done Done New true 237 Acked 1236 Read 5 Done Done New true 238 Acked 1237 Flush 0 Done Done New true 239 Acked 1238 Read 5 Done Done New true 240 Acked 1239 Flush 0 Done Done New true 241 Acked 1240 Flush 0 Done Done New true 242 Acked 1241 Read 5 Done Done New true 243 Acked 1242 Read 7 Done Done New true 244 Acked 1243 Write 2 Done Done New true 245 Acked 1244 Write 1 Done Done New true 246 Acked 1245 Write 5 Done Done New true 247 Acked 1246 Read 6 Done Done New true 248 Acked 1247 Write 2 Done Done New true 249 Acked 1248 Read 2 Done Done New true 250 Acked 1249 Write 9 Done Done New true 251 Acked 1250 Write 1 Done Done New true 252 Acked 1251 Write 10 Done Done New true 253 Acked 1252 Write 6 Done Done New true 254 Acked 1253 Write 2 Done Done New true 255 Acked 1254 Write 3 Done Done New true 256 Acked 1255 Read 9 Done Done New true 257 Acked 1256 Read 5 Done Done New true 258 Acked 1257 Read 2 Done Done New true 259 Acked 1258 Flush 0 Done Done New true 260 Acked 1259 Write 4 Done Done New true 261 Acked 1260 Flush 0 Done Done New true 262 Acked 1261 Read 7 Done Done New true 263 Acked 1262 Read 2 Done Done New true 264 Acked 1263 Write 2 Done Done New true 265 Acked 1264 Read 3 Done Done New true 266 Acked 1265 Flush 0 Done Done New true 267 Acked 1266 Read 6 Done Done New true 268 Acked 1267 Read 5 Done Done New true 269 Acked 1268 Write 5 Done Done New true 270 Acked 1269 Write 4 Done Done New true 271 Acked 1270 Write 5 Done Done New true 272 Acked 1271 Read 3 Done Done New true 273 Acked 1272 Write 4 Done Done New true 274 Acked 1273 Read 2 Done Done New true 275 Acked 1274 Write 2 Done Done New true 276 Acked 1275 Read 2 Done Done New true 277 Acked 1276 Write 7 Done Done New true 278 Acked 1277 Read 7 Done Done New true 279 Acked 1278 Read 3 Done Done New true 280 Acked 1279 Read 3 Done Done New true 281 Acked 1280 Write 1 Done Done New true 282 Acked 1281 Write 5 Done Done New true 283 Acked 1282 Write 7 Done Done New true 284 Acked 1283 Read 1 Done Done New true 285 Acked 1284 Write 3 Done Done New true 286 Acked 1285 Read 8 Done Done New true 287 Acked 1286 Read 5 Done Done New true 288 Acked 1287 Read 7 Done Done New true 289 Acked 1288 Read 7 Done Done New true 290 Acked 1289 Write 10 Done Done New true 291 Acked 1290 Read 1 Done Done New true 292 Acked 1291 Flush 0 Done Done New true 293 Acked 1292 Read 4 Done Done New true 294 Acked 1293 Write 8 Done Done New true 295 Acked 1294 Write 8 Done Done New true 296 Acked 1295 Write 1 Done Done New true 297 Acked 1296 Read 9 Done Done New true 298 Acked 1297 Read 5 Done Done New true 299 Acked 1298 Write 9 Done Done New true 300 Acked 1299 Write 10 Done Done New true 301 Acked 1300 Flush 0 Done Done New true STATES DS:0 DS:1 DS:2 TOTAL New 0 0 301 301 Sent 0 0 0 0 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1300 1300 0 Downstairs last five completed: Upstairs last five completed: 301 300 299 298 297 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.278636236Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.278861305Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.280854489Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.280885216Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.28090813Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 0","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.281248139Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.28128827Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.281330079Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.281371688Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:39.281412734Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1300 1299 1298 1297 1296 Upstairs last five completed: 301 300 299 298 297 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:30:43.159 INFO request completed, latency_us: 458, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 98ec8ddb-6737-496a-8a77-8fdfe53dd123, remote_addr: [8810] Got stop action so:false kr:false 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:43.161049892Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:43.161139562Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:43.161189408Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:43.161239881Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:43.161282635Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:43.16135322Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:30:43.161 INFO request completed, latency_us: 377, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2b41024e-791a-43cc-b0c3-5090ad091494, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:44.163151812Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:44.163218537Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:44.16324656Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:44.16327315Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:44.163293821Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:44.163408647Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:44.163453756Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:45.165874238Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:45.166211107Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:45.166372574Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:45.166850639Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:45.166885043Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:45.167218922Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:45.167833107Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:46.169225167Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:46.169290776Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:46.169318664Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:46.16934308Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:46.169369601Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:46.169390178Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:46.169412609Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:46.169454893Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:30:47.164 INFO request completed, latency_us: 242, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4f33cd6a-9071-4684-bc3a-f9e877ccf988, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 2719 len 28672 002/300 Flush 003/300 Read block 2412 len 36864 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:47.17087861Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:47.17095193Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:47.170981214Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:47.171885263Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:47.171929062Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 3 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:47.171966733Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:47.172263378Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 004/300 Flush 005/300 Read block 2973 len 16384 006/300 Write block 2974 len 24576 data: 2 2 2 2 2 2 007/300 Read block 950 len 12288 008/300 Read block 4216 len 28672 009/300 Flush 010/300 Read block 1732 len 4096 011/300 Write block 42 len 8192 data: 2 2 012/300 Read block 4158 len 28672 013/300 Write block 937 len 40960 data: 2 2 2 2 2 2 2 2 2 2 014/300 Flush 015/300 Read block 1305 len 8192 016/300 Read block 278 len 36864 017/300 Read block 4804 len 8192 018/300 Read block 1863 len 20480 019/300 Read block 957 len 24576 020/300 Write block 3666 len 12288 data: 2 2 2 021/300 Read block 1610 len 24576 022/300 Flush 023/300 Write block 1040 len 4096 data: 2 024/300 Write block 1256 len 12288 data: 2 2 2 025/300 Read block 2294 len 20480 026/300 Read block 856 len 40960 027/300 Read block 4128 len 20480 028/300 Read block 4448 len 20480 029/300 Read block 4248 len 20480 030/300 Flush 031/300 Flush 032/300 Write block 2269 len 36864 data: 2 2 2 2 2 2 2 2 2 033/300 Flush 034/300 Write block 4822 len 12288 data: 2 2 2 035/300 Read block 895 len 36864 036/300 Read block 4482 len 28672 037/300 Write block 1624 len 16384 data: 2 2 2 2 038/300 Read block 279 len 4096 039/300 Read block 2794 len 8192 040/300 Write block 2266 len 20480 data: 2 2 2 3 3 041/300 Write block 4187 len 32768 data: 2 2 2 2 2 2 2 2 042/300 Flush 043/300 Write block 3622 len 28672 data: 2 2 2 2 2 2 2 044/300 Write block 3678 len 4096 data: 2 045/300 Write block 4208 len 20480 data: 3 3 2 2 2 046/300 Read block 2731 len 32768 047/300 Write block 3300 len 28672 data: 2 2 2 2 2 2 2 048/300 Read block 3556 len 36864 049/300 Read block 3951 len 36864 050/300 Write block 2764 len 40960 data: 2 2 2 2 2 2 2 2 2 2 051/300 Read block 2681 len 32768 052/300 Write block 3629 len 40960 data: 2 2 2 2 2 2 2 2 2 2 053/300 Flush 054/300 Write block 2001 len 40960 data: 2 2 2 2 2 2 2 2 2 2 055/300 Write block 667 len 4096 data: 3 056/300 Write block 1591 len 28672 data: 2 2 2 2 2 2 2 057/300 Read block 187 len 36864 058/300 Read block 482 len 32768 059/300 Read block 792 len 24576 060/300 Read block 3964 len 24576 061/300 Flush 062/300 Read block 4622 len 36864 063/300 Read block 1322 len 36864 064/300 Read block 201 len 16384 065/300 Write block 1112 len 16384 data: 2 2 2 2 066/300 Write block 1392 len 20480 data: 3 3 2 2 2 067/300 Write block 3897 len 32768 data: 2 2 2 2 2 2 2 2 068/300 Read block 3324 len 40960 069/300 Write block 4422 len 32768 data: 2 3 3 2 2 2 2 2 070/300 Read block 4101 len 20480 071/300 Write block 892 len 36864 data: 3 3 3 3 2 2 2 2 2 072/300 Read block 44 len 8192 073/300 Flush 074/300 Read block 1754 len 4096 075/300 Write block 4724 len 36864 data: 3 3 3 3 3 3 2 2 2 076/300 Write block 3499 len 28672 data: 2 2 2 2 2 2 2 077/300 Read block 4392 len 16384 078/300 Read block 4178 len 4096 079/300 Flush 080/300 Write block 2613 len 12288 data: 2 2 2 081/300 Flush 082/300 Write block 1105 len 32768 data: 2 2 2 2 2 2 2 3 083/300 Flush 084/300 Read block 3068 len 36864 085/300 Write block 4761 len 32768 data: 2 2 2 2 2 3 3 3 086/300 Flush 087/300 Write block 3868 len 20480 data: 2 2 2 2 2 088/300 Write block 1177 len 4096 data: 2 089/300 Flush 090/300 Read block 2709 len 8192 091/300 Write block 2209 len 28672 data: 2 2 2 2 2 2 2 092/300 Read block 2257 len 20480 093/300 Read block 1719 len 20480 094/300 Read block 3073 len 24576 095/300 Write block 4866 len 16384 data: 2 2 2 2 096/300 Read block 1353 len 36864 097/300 Read block 534 len 32768 098/300 Read block 1034 len 36864 099/300 Read block 1074 len 4096 100/300 Write block 4449 len 8192 data: 2 2 101/300 Read block 422 len 12288 102/300 Read block 2954 len 32768 103/300 Read block 39 len 8192 104/300 Read block 4576 len 40960 105/300 Flush 106/300 Write block 4692 len 36864 data: 2 2 2 2 2 2 2 2 2 107/300 Read block 2926 len 16384 108/300 Read block 4457 len 20480 109/300 Write block 2791 len 12288 data: 2 2 2 110/300 Write block 1503 len 24576 data: 2 3 3 3 3 3 111/300 Write block 1986 len 24576 data: 2 2 2 2 2 2 112/300 Write block 1542 len 12288 data: 2 2 2 113/300 Write block 1751 len 36864 data: 2 2 2 2 2 2 2 2 2 114/300 Read block 3872 len 32768 115/300 Read block 226 len 12288 116/300 Write block 308 len 20480 data: 2 2 2 2 2 117/300 Write block 166 len 32768 data: 2 2 2 2 2 2 2 2 118/300 Read block 451 len 4096 119/300 Read block 2465 len 28672 120/300 Read block 849 len 40960 121/300 Flush 122/300 Read block 4662 len 36864 123/300 Read block 2751 len 36864 124/300 Read block 778 len 16384 125/300 Read block 3439 len 12288 126/300 Read block 487 len 24576 127/300 Write block 1325 len 12288 data: 2 2 2 128/300 Write block 1495 len 4096 data: 2 129/300 Read block 4719 len 20480 130/300 Read block 2396 len 12288 131/300 Flush 132/300 Write block 4041 len 12288 data: 2 2 2 133/300 Write block 3705 len 12288 data: 2 2 2 134/300 Write block 4805 len 24576 data: 2 2 2 2 2 2 135/300 Read block 3698 len 4096 136/300 Read block 921 len 36864 137/300 Write block 2819 len 20480 data: 2 2 2 2 2 138/300 Flush 139/300 Read block 1953 len 16384 140/300 Write block 2312 len 40960 data: 2 2 2 2 2 2 2 2 2 2 141/300 Write block 1221 len 12288 data: 2 2 2 142/300 Read block 3184 len 16384 143/300 Write block 2156 len 12288 data: 2 2 2 144/300 Write block 4164 len 28672 data: 2 2 2 2 2 2 2 145/300 Write block 950 len 20480 data: 2 2 2 2 2 146/300 Write block 3243 len 8192 data: 2 2 147/300 Read block 3267 len 16384 148/300 Read block 3089 len 8192 149/300 Write block 2828 len 4096 data: 2 150/300 Write block 2799 len 12288 data: 2 2 2 151/300 Write block 4244 len 28672 data: 2 2 2 2 3 3 3 152/300 Flush 153/300 Read block 4364 len 36864 154/300 Read block 3810 len 4096 155/300 Read block 1903 len 8192 156/300 Read block 4284 len 8192 157/300 Read block 3352 len 16384 158/300 Read block 2329 len 32768 159/300 Read block 3280 len 16384 160/300 Write block 4648 len 36864 data: 2 2 2 2 2 2 2 2 2 161/300 Write block 398 len 28672 data: 2 2 2 2 2 2 2 162/300 Write block 4485 len 4096 data: 2 163/300 Write block 2008 len 20480 data: 3 3 3 2 2 164/300 Read block 4178 len 40960 165/300 Write block 2406 len 28672 data: 2 2 2 2 2 2 2 166/300 Write block 4096 len 28672 data: 2 2 2 2 2 2 2 167/300 Flush 168/300 Flush 169/300 Flush 170/300 Write block 1963 len 12288 data: 2 2 2 171/300 Write block 2445 len 16384 data: 3 3 3 3 172/300 Read block 3965 len 4096 173/300 Read block 3043 len 20480 174/300 Flush 175/300 Read block 4569 len 12288 176/300 Read block 1244 len 20480 177/300 Write block 4503 len 32768 data: 2 2 2 2 2 2 2 2 178/300 Read block 927 len 4096 179/300 Read block 429 len 8192 180/300 Read block 2346 len 16384 181/300 Read block 3391 len 24576 182/300 Write block 2608 len 12288 data: 2 2 2 183/300 Read block 3204 len 8192 184/300 Write block 1166 len 20480 data: 2 2 2 2 2 185/300 Write block 3237 len 20480 data: 2 2 2 2 2 186/300 Flush 187/300 Read block 811 len 40960 188/300 Read block 3444 len 28672 189/300 Read block 4870 len 28672 190/300 Write block 3857 len 40960 data: 2 2 2 2 3 3 3 3 2 2 191/300 Write block 3178 len 28672 data: 2 2 2 2 2 2 2 192/300 Write block 2573 len 28672 data: 2 2 2 2 2 2 2 193/300 Write block 2176 len 4096 data: 2 194/300 Read block 3632 len 28672 195/300 Write block 409 len 20480 data: 2 2 2 2 2 196/300 Write block 3691 len 16384 data: 2 2 2 2 197/300 Write block 835 len 12288 data: 2 2 2 198/300 Write block 4927 len 28672 data: 3 3 2 2 2 2 2 199/300 Read block 1073 len 28672 200/300 Read block 4288 len 36864 201/300 Write block 4251 len 32768 data: 3 3 3 3 3 3 3 2 202/300 Write block 222 len 28672 data: 2 2 2 2 2 2 2 203/300 Read block 1455 len 40960 204/300 Read block 425 len 24576 205/300 Read block 3508 len 12288 206/300 Read block 2288 len 40960 207/300 Write block 2546 len 20480 data: 2 2 2 2 2 208/300 Write block 4733 len 4096 data: 3 209/300 Read block 825 len 16384 210/300 Write block 4113 len 8192 data: 2 2 211/300 Read block 1961 len 4096 212/300 Write block 2802 len 36864 data: 2 2 2 2 2 2 2 2 2 213/300 Flush 214/300 Write block 810 len 36864 data: 2 2 2 2 2 2 2 2 2 215/300 Read block 1965 len 36864 216/300 Flush 217/300 Write block 582 len 8192 data: 2 2 218/300 Flush 219/300 Write block 2764 len 36864 data: 3 3 3 3 3 3 3 3 3 220/300 Read block 1583 len 28672 221/300 Read block 804 len 12288 222/300 Write block 4842 len 4096 data: 4 223/300 Read block 4744 len 8192 224/300 Read block 2681 len 20480 225/300 Read block 2961 len 24576 226/300 Read block 1340 len 32768 227/300 Read block 2232 len 16384 228/300 Write block 2933 len 28672 data: 2 3 2 2 2 2 2 229/300 Read block 35 len 32768 230/300 Read block 1632 len 24576 231/300 Read block 3061 len 8192 232/300 Read block 267 len 20480 233/300 Write block 1949 len 4096 data: 2 234/300 Read block 2931 len 36864 235/300 Read block 949 len 40960 236/300 Write block 1141 len 4096 data: 2 237/300 Read block 2850 len 8192 238/300 Read block 2405 len 12288 239/300 Read block 3521 len 20480 240/300 Write block 3497 len 4096 data: 2 241/300 Read block 432 len 28672 242/300 Write block 2922 len 4096 data: 2 243/300 Read block 15 len 4096 244/300 Write block 2412 len 36864 data: 3 3 3 2 2 2 2 2 2 245/300 Write block 2356 len 40960 data: 2 2 2 2 2 2 2 2 2 2 246/300 Write block 3654 len 40960 data: 2 2 2 2 2 2 2 2 2 2 247/300 Read block 4036 len 12288 248/300 Write block 1204 len 36864 data: 2 2 2 2 2 2 2 2 2 249/300 Write block 3417 len 20480 data: 2 2 2 2 2 250/300 Write block 3947 len 40960 data: 2 2 2 2 2 2 2 2 2 2 251/300 Write block 1585 len 32768 data: 2 2 2 2 2 2 3 3 252/300 Flush 253/300 Read block 1191 len 12288 254/300 Read block 4773 len 40960 255/300 Read block 3957 len 40960 256/300 Read block 629 len 28672 257/300 Read block 796 len 4096 258/300 Read block 4053 len 24576 259/300 Read block 3603 len 36864 260/300 Write block 4304 len 4096 data: 2 261/300 Read block 4930 len 12288 262/300 Write block 2603 len 16384 data: 2 2 2 2 263/300 Read block 144 len 32768 264/300 Read block 3164 len 20480 265/300 Read block 4575 len 20480 266/300 Read block 1704 len 20480 267/300 Write block 1772 len 40960 data: 2 2 2 2 2 2 2 2 2 2 268/300 Write block 373 len 40960 data: 2 2 2 2 2 2 2 2 2 2 269/300 Read block 3949 len 28672 270/300 Read block 150 len 40960 271/300 Write block 1379 len 4096 data: 2 272/300 Read block 4226 len 16384 273/300 Read block 1216 len 4096 274/300 Read block 3823 len 8192 275/300 Read block 4549 len 20480 276/300 Write block 843 len 40960 data: 2 2 2 2 2 2 2 2 2 2 277/300 Write block 2765 len 36864 data: 4 4 4 4 4 4 4 4 3 278/300 Write block 2886 len 20480 data: 2 2 3 2 2 279/300 Flush 280/300 Flush 281/300 Write block 4324 len 28672 data: 2 2 2 2 2 2 2 282/300 Write block 279 len 36864 data: 2 2 2 2 2 2 2 2 2 283/300 Read block 2354 len 16384 284/300 Read block 55 len 24576 285/300 Read block 2507 len 36864 286/300 Read block 4169 len 28672 287/300 Read block 3438 len 28672 288/300 Read block 1010 len 28672 289/300 Write block 3359 len 40960 data: 2 2 2 2 2 2 2 2 2 2 290/300 Read block 76 len 16384 291/300 Write block 267 len 32768 data: 2 2 2 2 2 2 2 2 292/300 Write block 1181 len 4096 data: 2 293/300 Read block 861 len 20480 294/300 Read block 145 len 24576 295/300 Write block 996 len 32768 data: 2 2 2 2 2 2 2 2 296/300 Read block 4681 len 16384 297/300 Read block 1508 len 16384 298/300 Write block 2413 len 36864 data: 4 4 3 3 3 3 3 3 2 299/300 Read block 4702 len 28672 300/300 Write block 1108 len 16384 data: 3 3 3 3 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:30:48.102 INFO request completed, latency_us: 302, response_code: 204, uri: /start/cid/0, method: POST, req_id: 04a73cec-1682-4875-992f-55549999be97, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 302 Acked 1301 Read 7 New Done Done true 303 Acked 1302 Flush 0 New Done Done true 304 Acked 1303 Read 9 New Done Done true 305 Acked 1304 Flush 0 New Done Done false 306 Acked 1305 Read 4 New Done Done false 307 Acked 1306 Write 6 New Done Done false 308 Acked 1307 Read 3 New Done Done false 309 Acked 1308 Read 7 New Done Done false 310 Acked 1309 Flush 0 New Done Done false 311 Acked 1310 Read 1 New Done Done false 312 Acked 1311 Write 2 New Done Done false 313 Acked 1312 Read 7 New Done Done false 314 Acked 1313 Write 10 New Done Done false 315 Acked 1314 Flush 0 New Done Done false 316 Acked 1315 Read 2 New Done Done false 317 Acked 1316 Read 9 New Done Done false 318 Acked 1317 Read 2 New Done Done false 319 Acked 1318 Read 5 New Done Done false 320 Acked 1319 Read 6 New Done Done false 321 Acked 1320 Flush 0 New Done Done false 322 Acked 1321 Write 3 New Done Done false 323 Acked 1322 Read 6 New Done Done false 324 Acked 1323 Flush 0 New Done Done false 325 Acked 1324 Write 1 New Done Done false 326 Acked 1325 Write 3 New Done Done false 327 Acked 1326 Read 5 New Done Done false 328 Acked 1327 Read 10 New Done Done false 329 Acked 1328 Read 5 New Done Done false 330 Acked 1329 Read 5 New Done Done false 331 Acked 1330 Read 5 New Done Done false 332 Acked 1331 Flush 0 New Done Done false 333 Acked 1332 Flush 0 New Done Done false 334 Acked 1333 Write 9 New Done Done false 335 Acked 1334 Flush 0 New Done Done false 336 Acked 1335 Write 3 New Done Done false 337 Acked 1336 Read 9 New Done Done false 338 Acked 1337 Read 7 New Done Done false 339 Acked 1338 Write 4 New Done Done false 340 Acked 1339 Read 1 New Done Done false 341 Acked 1340 Read 2 New Done Done false 342 Acked 1341 Write 5 New Done Done false 343 Acked 1342 Write 8 New Done Done false 344 Acked 1343 Flush 0 New Done Done false 345 Acked 1344 Write 7 New Done Done false 346 Acked 1345 Write 1 New Done Done false 347 Acked 1346 Write 5 New Done Done false 348 Acked 1347 Read 8 New Done Done false 349 Acked 1348 Write 7 New Done Done false 350 Acked 1349 Read 9 New Done Done false 351 Acked 1350 Read 9 New Done Done false 352 Acked 1351 Write 10 New Done Done false 353 Acked 1352 Read 8 New Done Done false 354 Acked 1353 Write 10 New Done Done false 355 Acked 1354 Flush 0 New Done Done false 356 Acked 1355 Write 10 New Done Done false 357 Acked 1356 Write 1 New Done Done false 358 Acked 1357 Write 7 New Done Done false 359 Acked 1358 Read 9 New Done Done false 360 Acked 1359 Read 8 New Done Done false 361 Acked 1360 Read 6 New Done Done false 362 Acked 1361 Read 6 New Done Done false 363 Acked 1362 Flush 0 New Done Done false 364 Acked 1363 Read 9 New Done Done false 365 Acked 1364 Read 9 New Done Done false 366 Acked 1365 Read 4 New Done Done false 367 Acked 1366 Write 4 New Done Done false 368 Acked 1367 Write 5 New Done Done false 369 Acked 1368 Write 8 New Done Done false 370 Acked 1369 Read 10 New Done Done false 371 Acked 1370 Write 8 New Done Done false 372 Acked 1371 Read 5 New Done Done false 373 Acked 1372 Write 9 New Done Done false 374 Acked 1373 Read 2 New Done Done false 375 Acked 1374 Flush 0 New Done Done false 376 Acked 1375 Read 1 New Done Done false 377 Acked 1376 Write 9 New Done Done false 378 Acked 1377 Write 7 New Done Done false 379 Acked 1378 Read 4 New Done Done false 380 Acked 1379 Read 1 New Done Done false 381 Acked 1380 Flush 0 New Done Done false 382 Acked 1381 Write 3 New Done Done false 383 Acked 1382 Flush 0 New Done Done false 384 Acked 1383 Write 8 New Done Done false 385 Acked 1384 Flush 0 New Done Done false 386 Acked 1385 Read 9 New Done Done false 387 Acked 1386 Write 8 New Done Done false 388 Acked 1387 Flush 0 New Done Done false 389 Acked 1388 Write 5 New Done Done false 390 Acked 1389 Write 1 New Done Done false 391 Acked 1390 Flush 0 New Done Done false 392 Acked 1391 Read 2 New Done Done false 393 Acked 1392 Write 7 New Done Done false 394 Acked 1393 Read 5 New Done Done false 395 Acked 1394 Read 5 New Done Done false 396 Acked 1395 Read 6 New Done Done false 397 Acked 1396 Write 4 New Done Done false 398 Acked 1397 Read 9 New Done Done false 399 Acked 1398 Read 8 New Done Done false 400 Acked 1399 Read 9 New Done Done false 401 Acked 1400 Read 1 New Done Done false 402 Acked 1401 Write 2 New Done Done false 403 Acked 1402 Read 3 New Done Done false 404 Acked 1403 Read 8 New Done Done false 405 Acked 1404 Read 2 New Done Done false 406 Acked 1405 Read 10 New Done Done false 407 Acked 1406 Flush 0 New Done Done false 408 Acked 1407 Write 9 New Done Done false 409 Acked 1408 Read 4 New Done Done false 410 Acked 1409 Read 5 New Done Done false 411 Acked 1410 Write 3 New Done Done false 412 Acked 1411 Write 6 New Done Done false 413 Acked 1412 Write 6 New Done Done false 414 Acked 1413 Write 3 New Done Done false 415 Acked 1414 Write 9 New Done Done false 416 Acked 1415 Read 8 New Done Done false 417 Acked 1416 Read 3 New Done Done false 418 Acked 1417 Write 5 New Done Done false 419 Acked 1418 Write 8 New Done Done false 420 Acked 1419 Read 1 New Done Done false 421 Acked 1420 Read 7 New Done Done false 422 Acked 1421 Read 10 New Done Done false 423 Acked 1422 Flush 0 New Done Done false 424 Acked 1423 Read 9 New Done Done false 425 Acked 1424 Read 9 New Done Done false 426 Acked 1425 Read 4 New Done Done false 427 Acked 1426 Read 3 New Done Done false 428 Acked 1427 Read 6 New Done Done false 429 Acked 1428 Write 3 New Done Done false 430 Acked 1429 Write 1 New Done Done false 431 Acked 1430 Read 5 New Done Done false 432 Acked 1431 Read 3 New Done Done false 433 Acked 1432 Flush 0 New Done Done false 434 Acked 1433 Write 3 New Done Done false 435 Acked 1434 Write 3 New Done Done false 436 Acked 1435 Write 6 New Done Done false 437 Acked 1436 Read 1 New Done Done false 438 Acked 1437 Read 9 New Done Done false 439 Acked 1438 Write 5 New Done Done false 440 Acked 1439 Flush 0 New Done Done false 441 Acked 1440 Read 4 New Done Done false 442 Acked 1441 Write 10 New Done Done false 443 Acked 1442 Write 3 New Done Done false 444 Acked 1443 Read 4 New Done Done false 445 Acked 1444 Write 3 New Done Done false 446 Acked 1445 Write 7 New Done Done false 447 Acked 1446 Write 5 New Done Done false 448 Acked 1447 Write 2 New Done Done false 449 Acked 1448 Read 4 New Done Done false 450 Acked 1449 Read 2 New Done Done false 451 Acked 1450 Write 1 New Done Done false 452 Acked 1451 Write 3 New Done Done false 453 Acked 1452 Write 7 New Done Done false 454 Acked 1453 Flush 0 New Done Done false 455 Acked 1454 Read 9 New Done Done false 456 Acked 1455 Read 1 New Done Done false 457 Acked 1456 Read 2 New Done Done false 458 Acked 1457 Read 2 New Done Done false 459 Acked 1458 Read 4 New Done Done false 460 Acked 1459 Read 8 New Done Done false 461 Acked 1460 Read 4 New Done Done false 462 Acked 1461 Write 9 New Done Done false 463 Acked 1462 Write 7 New Done Done false 464 Acked 1463 Write 1 New Done Done false 465 Acked 1464 Write 5 New Done Done false 466 Acked 1465 Read 10 New Done Done false 467 Acked 1466 Write 7 New Done Done false 468 Acked 1467 Write 7 New Done Done false 469 Acked 1468 Flush 0 New Done Done false 470 Acked 1469 Flush 0 New Done Done false 471 Acked 1470 Flush 0 New Done Done false 472 Acked 1471 Write 3 New Done Done false 473 Acked 1472 Write 4 New Done Done false 474 Acked 1473 Read 1 New Done Done false 475 Acked 1474 Read 5 New Done Done false 476 Acked 1475 Flush 0 New Done Done false 477 Acked 1476 Read 3 New Done Done false 478 Acked 1477 Read 5 New Done Done false 479 Acked 1478 Write 8 New Done Done false 480 Acked 1479 Read 1 New Done Done false 481 Acked 1480 Read 2 New Done Done false 482 Acked 1481 Read 4 New Done Done false 483 Acked 1482 Read 6 New Done Done false 484 Acked 1483 Write 3 New Done Done false 485 Acked 1484 Read 2 New Done Done false 486 Acked 1485 Write 5 New Done Done false 487 Acked 1486 Write 5 New Done Done false 488 Acked 1487 Flush 0 New Done Done false 489 Acked 1488 Read 10 New Done Done false 490 Acked 1489 Read 7 New Done Done false 491 Acked 1490 Read 7 New Done Done false 492 Acked 1491 Write 10 New Done Done false 493 Acked 1492 Write 7 New Done Done false 494 Acked 1493 Write 7 New Done Done false 495 Acked 1494 Write 1 New Done Done false 496 Acked 1495 Read 7 New Done Done false 497 Acked 1496 Write 5 New Done Done false 498 Acked 1497 Write 4 New Done Done false 499 Acked 1498 Write 3 New Done Done false 500 Acked 1499 Write 7 New Done Done false 501 Acked 1500 Read 7 New Done Done false 502 Acked 1501 Read 9 New Done Done false 503 Acked 1502 Write 8 New Done Done false 504 Acked 1503 Write 7 New Done Done false 505 Acked 1504 Read 10 New Done Done false 506 Acked 1505 Read 6 New Done Done false 507 Acked 1506 Read 3 New Done Done false 508 Acked 1507 Read 10 New Done Done false 509 Acked 1508 Write 5 New Done Done false 510 Acked 1509 Write 1 New Done Done false 511 Acked 1510 Read 4 New Done Done false 512 Acked 1511 Write 2 New Done Done false 513 Acked 1512 Read 1 New Done Done false 514 Acked 1513 Write 9 New Done Done false 515 Acked 1514 Flush 0 New Done Done false 516 Acked 1515 Write 9 New Done Done false 517 Acked 1516 Read 9 New Done Done false 518 Acked 1517 Flush 0 New Done Done false 519 Acked 1518 Write 2 New Done Done false 520 Acked 1519 Flush 0 New Done Done false 521 Acked 1520 Write 9 New Done Done false 522 Acked 1521 Read 7 New Done Done false 523 Acked 1522 Read 3 New Done Done false 524 Acked 1523 Write 1 New Done Done false 525 Acked 1524 Read 2 New Done Done false 526 Acked 1525 Read 5 New Done Done false 527 Acked 1526 Read 6 New Done Done false 528 Acked 1527 Read 8 New Done Done false 529 Acked 1528 Read 4 New Done Done false 530 Acked 1529 Write 7 New Done Done false 531 Acked 1530 Read 8 New Done Done false 532 Acked 1531 Read 6 New Done Done false 533 Acked 1532 Read 2 New Done Done false 534 Acked 1533 Read 5 New Done Done false 535 Acked 1534 Write 1 New Done Done false 536 Acked 1535 Read 9 New Done Done false 537 Acked 1536 Read 10 New Done Done false 538 Acked 1537 Write 1 New Done Done false 539 Acked 1538 Read 2 New Done Done false 540 Acked 1539 Read 3 New Done Done false 541 Acked 1540 Read 5 New Done Done false 542 Acked 1541 Write 1 New Done Done false 543 Acked 1542 Read 7 New Done Done false 544 Acked 1543 Write 1 New Done Done false 545 Acked 1544 Read 1 New Done Done false 546 Acked 1545 Write 9 New Done Done false 547 Acked 1546 Write 10 New Done Done false 548 Acked 1547 Write 10 New Done Done false 549 Acked 1548 Read 3 New Done Done false 550 Acked 1549 Write 9 New Done Done false 551 Acked 1550 Write 5 New Done Done false 552 Acked 1551 Write 10 New Done Done false 553 Acked 1552 Write 8 New Done Done false 554 Acked 1553 Flush 0 New Done Done false 555 Acked 1554 Read 3 New Done Done false 556 Acked 1555 Read 10 New Done Done false 557 Acked 1556 Read 10 New Done Done false 558 Acked 1557 Read 7 New Done Done false 559 Acked 1558 Read 1 New Done Done false 560 Acked 1559 Read 6 New Done Done false 561 Acked 1560 Read 9 New Done Done false 562 Acked 1561 Write 1 New Done Done false 563 Acked 1562 Read 3 New Done Done false 564 Acked 1563 Write 4 New Done Done false 565 Acked 1564 Read 8 New Done Done false 566 Acked 1565 Read 5 New Done Done false 567 Acked 1566 Read 5 New Done Done false 568 Acked 1567 Read 5 New Done Done false 569 Acked 1568 Write 10 New Done Done false 570 Acked 1569 Write 10 New Done Done false 571 Acked 1570 Read 7 New Done Done false 572 Acked 1571 Read 10 New Done Done false 573 Acked 1572 Write 1 New Done Done false 574 Acked 1573 Read 4 New Done Done false 575 Acked 1574 Read 1 New Done Done false 576 Acked 1575 Read 2 New Done Done false 577 Acked 1576 Read 5 New Done Done false 578 Acked 1577 Write 10 New Done Done false 579 Acked 1578 Write 9 New Done Done false 580 Acked 1579 Write 5 New Done Done false 581 Acked 1580 Flush 0 New Done Done false 582 Acked 1581 Flush 0 New Done Done false 583 Acked 1582 Write 7 New Done Done false 584 Acked 1583 Write 9 New Done Done false 585 Acked 1584 Read 4 New Done Done false 586 Acked 1585 Read 6 New Done Done false 587 Acked 1586 Read 9 New Done Done false 588 Acked 1587 Read 7 New Done Done false 589 Acked 1588 Read 7 New Done Done false 590 Acked 1589 Read 7 New Done Done false 591 Acked 1590 Write 10 New Done Done false 592 Acked 1591 Read 4 New Done Done false 593 Acked 1592 Write 8 New Done Done false 594 Acked 1593 Write 1 New Done Done false 595 Acked 1594 Read 5 New Done Done false 596 Acked 1595 Read 6 New Done Done false 597 Acked 1596 Write 8 New Done Done false 598 Acked 1597 Read 4 New Done Done false 599 Acked 1598 Read 4 New Done Done false 600 Acked 1599 Write 9 New Done Done false 601 Acked 1600 Read 7 New Done Done false 602 Acked 1601 Write 4 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1300 1581 1581 Downstairs last five completed: 1300 1299 1298 1297 1296 Upstairs last five completed: 602 601 600 599 598 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:48.172821826Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:48.172886914Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:48.172915821Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:48.172942119Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:48.172962479Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:48.172985186Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:48.173210333Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:49.173433736Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:49.173502532Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:49.173531179Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:49.17355772Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:49.173578344Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:49.173674551Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:49.173923451Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1187) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.174817404Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.175031639Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.176983357Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.177019326Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.177042958Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 1300","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.177334545Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.17736849Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.177391724Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.177414125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:50.177436239Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1602 1601 1600 1599 1598 Upstairs last five completed: 603 602 601 600 599 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:30:52.106 INFO request completed, latency_us: 418, response_code: 204, uri: /stop/cid/0, [8810] Got stop action so:false kr:false method: POST, req_id: b66041b8-85ce-4551-bce2-3df181ce0386, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:30:52.107 INFO request completed, latency_us: 368, response_code: 200, uri: /state/cid/0, method: GET, req_id: e8a6ac6d-abb4-4185-9209-c48ff3adce07, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:52.107637547Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:52.107748795Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:52.107784972Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:52.107806861Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:52.107897004Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:52.1080134Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:53.109048196Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:53.109122733Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:53.10915108Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:53.109177826Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:53.10919804Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:53.109238788Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:53.109337184Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:54.111235937Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:54.111303729Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:54.111331675Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:54.111358208Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:54.111378546Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:54.111477783Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:54.111555591Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:55.112610791Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:55.11268456Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:55.1127134Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:55.112748712Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:55.112769971Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:55.112870755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:55.112921247Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:30:56.109 INFO request completed, latency_us: 286, response_code: 200, uri: /state/cid/0, method: GET, req_id: a967d57b-86f4-4e16-b56a-94f9a0634684, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 1561 len 28672 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:56.113427167Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:56.113490786Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:56.113518914Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:56.113545304Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:56.113565568Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 002/300{ Read block 3793 len 40960 "msg":"[0] client re-new 1 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:56.113602963Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:56.113820208Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Write block 94 len 32768 data: 2 2 2 2 2 2 2 2 004/300 Write block 1743 len 28672 data: 2 2 2 2 2 2 2 005/300 Read block 3675 len 20480 006/300 Read block 3814 len 8192 007/300 Write block 4131 len 16384 data: 2 2 2 2 008/300 Flush 009/300 Write block 4339 len 32768 data: 2 2 3 3 3 3 3 3 010/300 Write block 2864 len 32768 data: 3 3 3 3 3 3 3 3 011/300 Read block 458 len 20480 012/300 Write block 431 len 40960 data: 2 2 2 2 2 2 2 2 2 2 013/300 Write block 2158 len 12288 data: 3 2 2 014/300 Read block 322 len 32768 015/300 Write block 2488 len 12288 data: 2 2 2 016/300 Read block 3302 len 4096 017/300 Write block 231 len 24576 data: 2 2 2 2 2 2 018/300 Read block 1444 len 40960 019/300 Write block 1629 len 32768 data: 2 2 2 2 2 2 2 2 020/300 Read block 970 len 28672 021/300 Read block 736 len 24576 022/300 Read block 2204 len 24576 023/300 Write block 2271 len 16384 data: 3 3 3 3 024/300 Read block 2074 len 24576 025/300 Read block 2756 len 12288 026/300 Read block 3368 len 20480 027/300 Read block 339 len 32768 028/300 Read block 1725 len 12288 029/300 Read block 3269 len 12288 030/300 Read block 1786 len 12288 031/300 Write block 656 len 16384 data: 2 2 2 2 032/300 Write block 268 len 24576 data: 3 3 3 3 3 3 033/300 Write block 846 len 36864 data: 3 3 3 3 3 3 3 2 2 034/300 Write block 4695 len 16384 data: 3 3 3 3 035/300 Read block 1631 len 36864 036/300 Write block 2780 len 4096 data: 2 037/300 Read block 4760 len 20480 038/300 Read block 3276 len 32768 039/300 Read block 2017 len 32768 040/300 Read block 3603 len 12288 041/300 Write block 11 len 4096 data: 2 042/300 Flush 043/300 Read block 4737 len 8192 044/300 Read block 3595 len 4096 045/300 Read block 508 len 12288 046/300 Read block 3169 len 8192 047/300 Write block 3034 len 8192 data: 2 2 048/300 Write block 2714 len 40960 data: 2 2 2 2 2 2 2 2 2 2 049/300 Write block 2326 len 24576 data: 2 2 2 2 2 2 050/300 Read block 2884 len 40960 051/300 Write block 3025 len 36864 data: 2 2 2 2 2 2 2 2 2 052/300 Write block 643 len 24576 data: 2 2 2 2 2 2 053/300 Read block 18 len 4096 054/300 Write block 4045 len 32768 data: 2 2 2 2 2 2 2 2 055/300 Write block 2334 len 4096 data: 3 056/300 Write block 2483 len 28672 data: 3 2 2 2 2 3 3 057/300 Write block 10 len 36864 data: 2 3 2 2 2 2 2 2 2 058/300 Read block 2717 len 36864 059/300 Write block 4796 len 12288 data: 2 2 2 060/300 Read block 224 len 20480 061/300 Read block 1475 len 36864 062/300 Write block 4820 len 8192 data: 2 2 063/300 Read block 3869 len 40960 064/300 Write block 1655 len 40960 data: 2 2 2 2 2 2 2 2 2 2 065/300 Write block 1070 len 16384 data: 3 2 2 2 066/300 Read block 1334 len 36864 067/300 Write block 1223 len 40960 data: 3 2 2 2 2 2 2 2 2 2 068/300 Read block 3677 len 8192 069/300 Write block 597 len 40960 data: 2 2 2 2 2 2 2 2 2 2 070/300 Read block 171 len 28672 071/300 Write block 3186 len 32768 data: 2 2 2 2 2 2 2 2 072/300 Read block 3921 len 24576 073/300 Read block 3269 len 24576 074/300 Flush 075/300 Write block 4383 len 36864 data: 2 2 2 2 2 3 3 2 2 076/300 Flush 077/300 Write block 460 len 16384 data: 2 2 2 2 078/300 Write block 687 len 12288 data: 2 2 2 079/300 Read block 2753 len 36864 080/300 Read block 81 len 8192 081/300 Read block 4044 len 32768 082/300 Read block 3484 len 20480 083/300 Flush 084/300 Write block 4317 len 28672 data: 2 2 2 2 2 2 2 085/300 Read block 2277 len 24576 086/300 Write block 963 len 16384 data: 2 2 2 2 087/300 Write block 4501 len 36864 data: 2 2 3 3 3 3 3 3 3 088/300 Read block 1687 len 32768 089/300 Write block 4080 len 36864 data: 2 2 2 2 2 2 2 2 2 090/300 Read block 2727 len 16384 091/300 Read block 3238 len 40960 092/300 Read block 2208 len 28672 093/300 Read block 3729 len 8192 094/300 Write block 1099 len 12288 data: 2 2 2 095/300 Read block 53 len 32768 096/300 Write block 4983 len 28672 data: 3 3 2 2 2 2 2 097/300 Write block 1106 len 28672 data: 3 3 4 4 4 4 4 098/300 Read block 3155 len 36864 099/300 Read block 4722 len 12288 100/300 Read block 1796 len 32768 101/300 Write block 3251 len 40960 data: 2 2 2 2 2 2 2 2 3 3 102/300 Read block 4640 len 8192 103/300 Write block 2373 len 4096 data: 2 104/300 Write block 2961 len 8192 data: 3 3 105/300 Read block 1767 len 32768 106/300 Write block 4633 len 36864 data: 2 2 2 2 2 2 2 2 2 107/300 Read block 847 len 20480 108/300 Read block 686 len 40960 109/300 Read block 3648 len 20480 110/300 Read block 836 len 32768 111/300 Write block 14 len 24576 data: 3 3 3 3 3 2 112/300 Write block 2373 len 8192 data: 3 2 113/300 Read block 3468 len 40960 114/300 Write block 1977 len 12288 data: 2 2 2 115/300 Read block 62 len 32768 116/300 Read block 2665 len 4096 117/300 Read block 578 len 20480 118/300 Flush 119/300 Read block 1307 len 32768 120/300 Read block 499 len 32768 121/300 Flush 122/300 Write block 2824 len 20480 data: 2 2 2 2 3 123/300 Flush 124/300 Read block 903 len 16384 125/300 Read block 2310 len 24576 126/300 Read block 4330 len 12288 127/300 Write block 451 len 36864 data: 2 2 3 2 2 2 2 2 2 128/300 Read block 1345 len 20480 129/300 Read block 4439 len 36864 130/300 Read block 4432 len 4096 131/300 Read block 4509 len 36864 132/300 Write block 1387 len 4096 data: 2 133/300 Flush 134/300 Read block 4583 len 4096 135/300 Write block 3975 len 28672 data: 2 2 2 2 2 2 2 136/300 Read block 4178 len 16384 137/300 Read block 2666 len 16384 138/300 Write block 3788 len 20480 data: 3 3 3 3 3 139/300 Write block 4046 len 16384 data: 3 3 3 3 140/300 Write block 772 len 12288 data: 2 2 2 141/300 Read block 4797 len 16384 142/300 Write block 4498 len 12288 data: 2 2 2 143/300 Write block 3736 len 20480 data: 2 2 2 2 2 144/300 Write block 2438 len 8192 data: 2 2 145/300 Write block 4324 len 28672 data: 3 3 3 3 3 3 3 146/300 Write block 3147 len 36864 data: 2 2 2 2 2 2 2 2 2 147/300 Read block 3290 len 16384 148/300 Flush 149/300 Flush 150/300 Write block 48 len 36864 data: 4 4 4 3 3 3 3 3 2 151/300 Write block 847 len 8192 data: 4 4 152/300 Read block 322 len 8192 153/300 Read block 6 len 24576 154/300 Read block 2705 len 16384 155/300 Read block 1827 len 12288 156/300 Read block 3113 len 32768 157/300 Write block 4645 len 16384 data: 3 2 2 3 158/300 Write block 191 len 32768 data: 2 2 2 2 2 2 2 2 159/300 Read block 4432 len 28672 160/300 Read block 4806 len 20480 161/300 Write block 2733 len 40960 data: 2 2 2 2 2 2 2 2 2 2 162/300 Read block 1490 len 36864 163/300 Read block 4729 len 4096 164/300 Read block 759 len 16384 165/300 Write block 3906 len 40960 data: 2 2 2 2 2 2 2 2 2 2 166/300 Write block 75 len 28672 data: 2 2 2 2 2 2 2 167/300 Read block 3959 len 12288 168/300 Read block 1045 len 8192 169/300 Write block 358 len 40960 data: 2 2 2 2 2 2 2 2 3 3 170/300 Write block 1274 len 16384 data: 2 2 3 3 171/300 Read block 3317 len 32768 172/300 Read block 4874 len 28672 173/300 Read block 1198 len 16384 174/300 Read block 751 len 28672 175/300 Read block 1827 len 36864 176/300 Write block 4442 len 28672 data: 3 3 3 2 3 3 3 177/300 Write block 1710 len 8192 data: 2 2 178/300 Read block 2083 len 20480 179/300 Read block 137 len 12288 180/300 Read block 846 len 4096 181/300 Write block 2769 len 24576 data: 5 5 5 5 4 2 182/300 Write block 897 len 28672 data: 3 3 3 3 2 2 2 183/300 Read block 3311 len 28672 184/300 Write block 3616 len 36864 data: 2 2 2 2 2 2 3 3 3 185/300 Write block 949 len 24576 data: 2 3 3 3 3 3 186/300 Read block 3099 len 4096 187/300 Flush 188/300 Read block 1045 len 24576 189/300 Read block 4195 len 40960 190/300 Read block 724 len 4096 191/300 Read block 99 len 12288 192/300 Write block 2542 len 8192 data: 2 2 193/300 Write block 1389 len 20480 data: 3 3 3 4 4 194/300 Read block 3159 len 32768 195/300 Write block 4797 len 40960 data: 3 3 2 2 2 2 2 2 3 3 196/300 Read block 4007 len 36864 197/300 Read block 1655 len 32768 198/300 Read block 1776 len 20480 199/300 Read block 775 len 28672 200/300 Write block 3080 len 16384 data: 2 2 2 2 201/300 Read block 1305 len 40960 202/300 Write block 4879 len 40960 data: 2 2 2 2 2 2 2 2 2 2 203/300 Write block 4842 len 20480 data: 5 3 3 2 2 204/300 Read block 731 len 32768 205/300 Write block 4637 len 4096 data: 3 206/300 Write block 1655 len 28672 data: 3 3 3 3 3 3 3 207/300 Flush 208/300 Read block 1260 len 32768 209/300 Flush 210/300 Flush 211/300 Flush 212/300 Flush 213/300 Write block 4318 len 40960 data: 3 3 3 3 3 3 4 4 4 4 214/300 Write block 2056 len 8192 data: 2 2 215/300 Write block 391 len 20480 data: 2 3 3 3 3 216/300 Write block 4931 len 32768 data: 3 3 3 2 2 2 2 2 217/300 Read block 251 len 32768 218/300 Read block 1962 len 28672 219/300 Read block 1800 len 24576 220/300 Write block 129 len 4096 data: 2 221/300 Write block 2513 len 8192 data: 2 2 222/300 Read block 1223 len 12288 223/300 Flush 224/300 Write block 599 len 16384 data: 3 3 3 3 225/300 Write block 132 len 20480 data: 2 2 2 2 2 226/300 Write block 188 len 36864 data: 2 2 2 3 3 3 3 3 3 227/300 Read block 1864 len 8192 228/300 Write block 1392 len 20480 data: 5 5 3 3 3 229/300 Read block 3184 len 32768 230/300 Write block 4710 len 4096 data: 2 231/300 Read block 2075 len 4096 232/300 Write block 103 len 16384 data: 2 2 2 2 233/300 Write block 2871 len 16384 data: 4 2 2 2 234/300 Write block 1714 len 8192 data: 3 3 235/300 Read block 2651 len 8192 236/300 Read block 1176 len 16384 237/300 Write block 2896 len 32768 data: 2 2 2 2 2 2 2 2 238/300 Write block 2857 len 16384 data: 2 2 2 2 239/300 Read block 410 len 40960 240/300 Write block 3722 len 20480 data: 2 3 3 3 3 241/300 Write block 3332 len 8192 data: 2 2 242/300 Read block 3568 len 40960 243/300 Write block 4155 len 12288 data: 2 2 2 244/300 Read block 3364 len 4096 245/300 Read block 1413 len 36864 246/300 Write block 3332 len 4096 data: 3 247/300 Write block 2119 len 24576 data: 2 2 2 2 2 2 248/300 Flush 249/300 Read block 3399 len 4096 250/300 Write block 4730 len 36864 data: 3 3 3 4 3 3 3 3 3 251/300 Read block 3961 len 24576 252/300 Read block 2656 len 8192 253/300 Read block 3770 len 32768 254/300 Write block 3240 len 40960 data: 3 3 2 3 3 2 2 2 2 2 255/300 Flush 256/300 Write block 1441 len 40960 data: 2 2 2 2 2 2 2 2 2 2 257/300 Write block 3602 len 40960 data: 2 2 2 2 2 2 2 2 2 2 258/300 Read block 877 len 4096 259/300 Read block 2469 len 8192 260/300 Read block 1701 len 32768 261/300 Write block 1743 len 20480 data: 3 3 3 3 3 262/300 Read block 3132 len 36864 263/300 Read block 378 len 16384 264/300 Read block 2109 len 12288 265/300 Write block 1131 len 32768 data: 2 3 3 3 3 3 3 3 266/300 Read block 476 len 24576 267/300 Flush 268/300 Write block 4895 len 40960 data: 2 2 2 2 2 2 2 3 3 3 269/300 Read block 2136 len 4096 270/300 Read block 4739 len 20480 271/300 Write block 3652 len 40960 data: 2 2 3 3 3 3 3 3 3 3 272/300 Read block 3525 len 36864 273/300 Read block 4788 len 16384 274/300 Flush 275/300 Write block 4379 len 32768 data: 2 2 2 2 3 3 3 3 276/300 Read block 133 len 36864 277/300 Write block 1226 len 24576 data: 3 3 3 3 3 3 278/300 Write block 3993 len 24576 data: 2 3 3 3 3 2 279/300 Read block 4903 len 4096 280/300 Write block 2358 len 40960 data: 3 3 3 3 3 3 3 3 2 2 281/300 Write block 238 len 8192 data: 2 2 282/300 Read block 2494 len 4096 283/300 Read block 1598 len 8192 284/300 Write block 966 len 40960 data: 3 2 2 2 2 2 3 3 3 3 285/300 Write block 2047 len 8192 data: 3 3 286/300 Write block 4402 len 8192 data: 3 2 287/300 Read block 554 len 24576 288/300 Read block 2237 len 16384 289/300 Write block 2399 len 28672 data: 2 2 2 2 2 2 2 290/300 Flush 291/300 Write block 3667 len 28672 data: 3 3 2 2 2 2 2 292/300 Read block 803 len 24576 293/300 Read block 3258 len 8192 294/300 Read block 1823 len 36864 295/300 Read block 1081 len 32768 296/300 Flush 297/300 Read block 2629 len 12288 298/300 Write block 2196 len 4096 data: 2 299/300 Flush 300/300 Read block 4250 len 8192 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:30:57.058 INFO request completed, latency_us: 283, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7ae8b57d-f040-4cc2-8af4-78fe8401bacf, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 604 Acked 1603 Read 7 New Done Done true 605 Acked 1604 Read 10 New Done Done false 606 Acked 1605 Write 8 New Done Done false 607 Acked 1606 Write 7 New Done Done false 608 Acked 1607 Read 5 New Done Done false 609 Acked 1608 Read 2 New Done Done false 610 Acked 1609 Write 4 New Done Done false 611 Acked 1610 Flush 0 New Done Done false 612 Acked 1611 Write 8 New Done Done false 613 Acked 1612 Write 8 New Done Done false 614 Acked 1613 Read 5 New Done Done false 615 Acked 1614 Write 10 New Done Done false 616 Acked 1615 Write 3 New Done Done false 617 Acked 1616 Read 8 New Done Done false 618 Acked 1617 Write 3 New Done Done false 619 Acked 1618 Read 1 New Done Done false 620 Acked 1619 Write 6 New Done Done false 621 Acked 1620 Read 10 New Done Done false 622 Acked 1621 Write 8 New Done Done false 623 Acked 1622 Read 7 New Done Done false 624 Acked 1623 Read 6 New Done Done false 625 Acked 1624 Read 6 New Done Done false 626 Acked 1625 Write 4 New Done Done false 627 Acked 1626 Read 6 New Done Done false 628 Acked 1627 Read 3 New Done Done false 629 Acked 1628 Read 5 New Done Done false 630 Acked 1629 Read 8 New Done Done false 631 Acked 1630 Read 3 New Done Done false 632 Acked 1631 Read 3 New Done Done false 633 Acked 1632 Read 3 New Done Done false 634 Acked 1633 Write 4 New Done Done false 635 Acked 1634 Write 6 New Done Done false 636 Acked 1635 Write 9 New Done Done false 637 Acked 1636 Write 4 New Done Done false 638 Acked 1637 Read 9 New Done Done false 639 Acked 1638 Write 1 New Done Done false 640 Acked 1639 Read 5 New Done Done false 641 Acked 1640 Read 8 New Done Done false 642 Acked 1641 Read 8 New Done Done false 643 Acked 1642 Read 3 New Done Done false 644 Acked 1643 Write 1 New Done Done false 645 Acked 1644 Flush 0 New Done Done false 646 Acked 1645 Read 2 New Done Done false 647 Acked 1646 Read 1 New Done Done false 648 Acked 1647 Read 3 New Done Done false 649 Acked 1648 Read 2 New Done Done false 650 Acked 1649 Write 2 New Done Done false 651 Acked 1650 Write 10 New Done Done false 652 Acked 1651 Write 6 New Done Done false 653 Acked 1652 Read 10 New Done Done false 654 Acked 1653 Write 9 New Done Done false 655 Acked 1654 Write 6 New Done Done false 656 Acked 1655 Read 1 New Done Done false 657 Acked 1656 Write 8 New Done Done false 658 Acked 1657 Write 1 New Done Done false 659 Acked 1658 Write 7 New Done Done false 660 Acked 1659 Write 9 New Done Done false 661 Acked 1660 Read 9 New Done Done false 662 Acked 1661 Write 3 New Done Done false 663 Acked 1662 Read 5 New Done Done false 664 Acked 1663 Read 9 New Done Done false 665 Acked 1664 Write 2 New Done Done false 666 Acked 1665 Read 10 New Done Done false 667 Acked 1666 Write 10 New Done Done false 668 Acked 1667 Write 4 New Done Done false 669 Acked 1668 Read 9 New Done Done false 670 Acked 1669 Write 10 New Done Done false 671 Acked 1670 Read 2 New Done Done false 672 Acked 1671 Write 10 New Done Done false 673 Acked 1672 Read 7 New Done Done false 674 Acked 1673 Write 8 New Done Done false 675 Acked 1674 Read 6 New Done Done false 676 Acked 1675 Read 6 New Done Done false 677 Acked 1676 Flush 0 New Done Done false 678 Acked 1677 Write 9 New Done Done false 679 Acked 1678 Flush 0 New Done Done false 680 Acked 1679 Write 4 New Done Done false 681 Acked 1680 Write 3 New Done Done false 682 Acked 1681 Read 9 New Done Done false 683 Acked 1682 Read 2 New Done Done false 684 Acked 1683 Read 8 New Done Done false 685 Acked 1684 Read 5 New Done Done false 686 Acked 1685 Flush 0 New Done Done false 687 Acked 1686 Write 7 New Done Done false 688 Acked 1687 Read 6 New Done Done false 689 Acked 1688 Write 4 New Done Done false 690 Acked 1689 Write 9 New Done Done false 691 Acked 1690 Read 8 New Done Done false 692 Acked 1691 Write 9 New Done Done false 693 Acked 1692 Read 4 New Done Done false 694 Acked 1693 Read 10 New Done Done false 695 Acked 1694 Read 7 New Done Done false 696 Acked 1695 Read 2 New Done Done false 697 Acked 1696 Write 3 New Done Done false 698 Acked 1697 Read 8 New Done Done false 699 Acked 1698 Write 7 New Done Done false 700 Acked 1699 Write 7 New Done Done false 701 Acked 1700 Read 9 New Done Done false 702 Acked 1701 Read 3 New Done Done false 703 Acked 1702 Read 8 New Done Done false 704 Acked 1703 Write 10 New Done Done false 705 Acked 1704 Read 2 New Done Done false 706 Acked 1705 Write 1 New Done Done false 707 Acked 1706 Write 2 New Done Done false 708 Acked 1707 Read 8 New Done Done false 709 Acked 1708 Write 9 New Done Done false 710 Acked 1709 Read 5 New Done Done false 711 Acked 1710 Read 10 New Done Done false 712 Acked 1711 Read 5 New Done Done false 713 Acked 1712 Read 8 New Done Done false 714 Acked 1713 Write 6 New Done Done false 715 Acked 1714 Write 2 New Done Done false 716 Acked 1715 Read 10 New Done Done false 717 Acked 1716 Write 3 New Done Done false 718 Acked 1717 Read 8 New Done Done false 719 Acked 1718 Read 1 New Done Done false 720 Acked 1719 Read 5 New Done Done false 721 Acked 1720 Flush 0 New Done Done false 722 Acked 1721 Read 8 New Done Done false 723 Acked 1722 Read 8 New Done Done false 724 Acked 1723 Flush 0 New Done Done false 725 Acked 1724 Write 5 New Done Done false 726 Acked 1725 Flush 0 New Done Done false 727 Acked 1726 Read 4 New Done Done false 728 Acked 1727 Read 6 New Done Done false 729 Acked 1728 Read 3 New Done Done false 730 Acked 1729 Write 9 New Done Done false 731 Acked 1730 Read 5 New Done Done false 732 Acked 1731 Read 9 New Done Done false 733 Acked 1732 Read 1 New Done Done false 734 Acked 1733 Read 9 New Done Done false 735 Acked 1734 Write 1 New Done Done false 736 Acked 1735 Flush 0 New Done Done false 737 Acked 1736 Read 1 New Done Done false 738 Acked 1737 Write 7 New Done Done false 739 Acked 1738 Read 4 New Done Done false 740 Acked 1739 Read 4 New Done Done false 741 Acked 1740 Write 5 New Done Done false 742 Acked 1741 Write 4 New Done Done false 743 Acked 1742 Write 3 New Done Done false 744 Acked 1743 Read 4 New Done Done false 745 Acked 1744 Write 3 New Done Done false 746 Acked 1745 Write 5 New Done Done false 747 Acked 1746 Write 2 New Done Done false 748 Acked 1747 Write 7 New Done Done false 749 Acked 1748 Write 9 New Done Done false 750 Acked 1749 Read 4 New Done Done false 751 Acked 1750 Flush 0 New Done Done false 752 Acked 1751 Flush 0 New Done Done false 753 Acked 1752 Write 9 New Done Done false 754 Acked 1753 Write 2 New Done Done false 755 Acked 1754 Read 2 New Done Done false 756 Acked 1755 Read 6 New Done Done false 757 Acked 1756 Read 4 New Done Done false 758 Acked 1757 Read 3 New Done Done false 759 Acked 1758 Read 8 New Done Done false 760 Acked 1759 Write 4 New Done Done false 761 Acked 1760 Write 8 New Done Done false 762 Acked 1761 Read 7 New Done Done false 763 Acked 1762 Read 5 New Done Done false 764 Acked 1763 Write 10 New Done Done false 765 Acked 1764 Read 9 New Done Done false 766 Acked 1765 Read 1 New Done Done false 767 Acked 1766 Read 4 New Done Done false 768 Acked 1767 Write 10 New Done Done false 769 Acked 1768 Write 7 New Done Done false 770 Acked 1769 Read 3 New Done Done false 771 Acked 1770 Read 2 New Done Done false 772 Acked 1771 Write 10 New Done Done false 773 Acked 1772 Write 4 New Done Done false 774 Acked 1773 Read 8 New Done Done false 775 Acked 1774 Read 7 New Done Done false 776 Acked 1775 Read 4 New Done Done false 777 Acked 1776 Read 7 New Done Done false 778 Acked 1777 Read 9 New Done Done false 779 Acked 1778 Write 7 New Done Done false 780 Acked 1779 Write 2 New Done Done false 781 Acked 1780 Read 5 New Done Done false 782 Acked 1781 Read 3 New Done Done false 783 Acked 1782 Read 1 New Done Done false 784 Acked 1783 Write 6 New Done Done false 785 Acked 1784 Write 7 New Done Done false 786 Acked 1785 Read 7 New Done Done false 787 Acked 1786 Write 9 New Done Done false 788 Acked 1787 Write 6 New Done Done false 789 Acked 1788 Read 1 New Done Done false 790 Acked 1789 Flush 0 New Done Done false 791 Acked 1790 Read 6 New Done Done false 792 Acked 1791 Read 10 New Done Done false 793 Acked 1792 Flush 0 New Done Done false 794 Acked 1793 Read 1 New Done Done false 795 Acked 1794 Read 3 New Done Done false 796 Acked 1795 Write 2 New Done Done false 797 Acked 1796 Write 5 New Done Done false 798 Acked 1797 Read 8 New Done Done false 799 Acked 1798 Write 10 New Done Done false 800 Acked 1799 Read 9 New Done Done false 801 Acked 1800 Read 8 New Done Done false 802 Acked 1801 Read 5 New Done Done false 803 Acked 1802 Read 7 New Done Done false 804 Acked 1803 Write 4 New Done Done false 805 Acked 1804 Read 10 New Done Done false 806 Acked 1805 Write 10 New Done Done false 807 Acked 1806 Write 5 New Done Done false 808 Acked 1807 Read 8 New Done Done false 809 Acked 1808 Write 1 New Done Done false 810 Acked 1809 Write 7 New Done Done false 811 Acked 1810 Flush 0 New Done Done false 812 Acked 1811 Read 8 New Done Done false 813 Acked 1812 Flush 0 New Done Done false 814 Acked 1813 Flush 0 New Done Done false 815 Acked 1814 Flush 0 New Done Done false 816 Acked 1815 Flush 0 New Done Done false 817 Acked 1816 Write 10 New Done Done false 818 Acked 1817 Write 2 New Done Done false 819 Acked 1818 Write 5 New Done Done false 820 Acked 1819 Write 8 New Done Done false 821 Acked 1820 Read 8 New Done Done false 822 Acked 1821 Read 7 New Done Done false 823 Acked 1822 Read 6 New Done Done false 824 Acked 1823 Write 1 New Done Done false 825 Acked 1824 Write 2 New Done Done false 826 Acked 1825 Read 3 New Done Done false 827 Acked 1826 Flush 0 New Done Done false 828 Acked 1827 Write 4 New Done Done false 829 Acked 1828 Write 5 New Done Done false 830 Acked 1829 Write 9 New Done Done false 831 Acked 1830 Read 2 New Done Done false 832 Acked 1831 Write 5 New Done Done false 833 Acked 1832 Read 8 New Done Done false 834 Acked 1833 Write 1 New Done Done false 835 Acked 1834 Read 1 New Done Done false 836 Acked 1835 Write 4 New Done Done false 837 Acked 1836 Write 4 New Done Done false 838 Acked 1837 Write 2 New Done Done false 839 Acked 1838 Read 2 New Done Done false 840 Acked 1839 Read 4 New Done Done false 841 Acked 1840 Write 8 New Done Done false 842 Acked 1841 Write 4 New Done Done false 843 Acked 1842 Read 10 New Done Done false 844 Acked 1843 Write 5 New Done Done false 845 Acked 1844 Write 2 New Done Done false 846 Acked 1845 Read 10 New Done Done false 847 Acked 1846 Write 3 New Done Done false 848 Acked 1847 Read 1 New Done Done false 849 Acked 1848 Read 9 New Done Done false 850 Acked 1849 Write 1 New Done Done false 851 Acked 1850 Write 6 New Done Done false 852 Acked 1851 Flush 0 New Done Done false 853 Acked 1852 Read 1 New Done Done false 854 Acked 1853 Write 9 New Done Done false 855 Acked 1854 Read 6 New Done Done false 856 Acked 1855 Read 2 New Done Done false 857 Acked 1856 Read 8 New Done Done false 858 Acked 1857 Write 10 New Done Done false 859 Acked 1858 Flush 0 New Done Done false 860 Acked 1859 Write 10 New Done Done false 861 Acked 1860 Write 10 New Done Done false 862 Acked 1861 Read 1 New Done Done false 863 Acked 1862 Read 2 New Done Done false 864 Acked 1863 Read 8 New Done Done false 865 Acked 1864 Write 5 New Done Done false 866 Acked 1865 Read 9 New Done Done false 867 Acked 1866 Read 4 New Done Done false 868 Acked 1867 Read 3 New Done Done false 869 Acked 1868 Write 8 New Done Done false 870 Acked 1869 Read 6 New Done Done false 871 Acked 1870 Flush 0 New Done Done false 872 Acked 1871 Write 10 New Done Done false 873 Acked 1872 Read 1 New Done Done false 874 Acked 1873 Read 5 New Done Done false 875 Acked 1874 Write 10 New Done Done false 876 Acked 1875 Read 9 New Done Done false 877 Acked 1876 Read 4 New Done Done false 878 Acked 1877 Flush 0 New Done Done false 879 Acked 1878 Write 8 New Done Done false 880 Acked 1879 Read 9 New Done Done false 881 Acked 1880 Write 6 New Done Done false 882 Acked 1881 Write 6 New Done Done false 883 Acked 1882 Read 1 New Done Done false 884 Acked 1883 Write 10 New Done Done false 885 Acked 1884 Write 2 New Done Done false 886 Acked 1885 Read 1 New Done Done false 887 Acked 1886 Read 2 New Done Done false 888 Acked 1887 Write 10 New Done Done false 889 Acked 1888 Write 2 New Done Done false 890 Acked 1889 Write 2 New Done Done false 891 Acked 1890 Read 6 New Done Done false 892 Acked 1891 Read 4 New Done Done false 893 Acked 1892 Write 7 New Done Done false 894 Acked 1893 Flush 0 New Done Done false 895 Acked 1894 Write 7 New Done Done false 896 Acked 1895 Read 6 New Done Done false 897 Acked 1896 Read 2 New Done Done false 898 Acked 1897 Read 9 New Done Done false 899 Acked 1898 Read 8 New Done Done false 900 Acked 1899 Flush 0 New Done Done false 901 Acked 1900 Read 3 New Done Done false 902 Acked 1901 Write 1 New Done Done false 903 Acked 1902 Flush 0 New Done Done false 904 Acked 1903 Read 2 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1602 1902 1902 Downstairs last five completed: 1602 1601 1600 1599 1598 Upstairs last five completed: 904 903 902 901 900 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:57.11489481Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:57.114958894Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:57.114987726Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:57.115011949Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:57.115038306Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:57.115058567Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:57.11508106Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:57.115319743Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:58.115727739Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:58.115857339Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:58.115885909Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:58.115912617Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:58.115932944Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:58.115973677Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:58.116246789Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:59.11644738Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:59.116511033Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:59.116539446Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:30:59.116565986Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:59.116586165Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:59.116629879Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:30:59.116885787Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:00.117689583Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:00.117762059Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:00.117790889Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:00.117817713Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:00.117838146Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:00.11788355Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:00.118141763Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1188) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 604 Acked 1603 Read 7 New Done Done true 605 Acked 1604 Read 10 New Done Done true 606 Acked 1605 Write 8 New Done Done true 607 Acked 1606 Write 7 New Done Done true 608 Acked 1607 Read 5 New Done Done true 609 Acked 1608 Read 2 New Done Done true 610 Acked 1609 Write 4 New Done Done true 611 Acked 1610 Flush 0 New Done Done true 612 Acked 1611 Write 8 New Done Done true 613 Acked 1612 Write 8 New Done Done true 614 Acked 1613 Read 5 New Done Done true 615 Acked 1614 Write 10 New Done Done true 616 Acked 1615 Write 3 New Done Done true 617 Acked 1616 Read 8 New Done Done true 618 Acked 1617 Write 3 New Done Done true 619 Acked 1618 Read 1 New Done Done true 620 Acked 1619 Write 6 New Done Done true 621 Acked 1620 Read 10 New Done Done true 622 Acked 1621 Write 8 New Done Done true 623 Acked 1622 Read 7 New Done Done true 624 Acked 1623 Read 6 New Done Done true 625 Acked 1624 Read 6 New Done Done true 626 Acked 1625 Write 4 New Done Done true 627 Acked 1626 Read 6 New Done Done true 628 Acked 1627 Read 3 New Done Done true 629 Acked 1628 Read 5 New Done Done true 630 Acked 1629 Read 8 New Done Done true 631 Acked 1630 Read 3 New Done Done true 632 Acked 1631 Read 3 New Done Done true 633 Acked 1632 Read 3 New Done Done true 634 Acked 1633 Write 4 New Done Done true 635 Acked 1634 Write 6 New Done Done true 636 Acked 1635 Write 9 New Done Done true 637 Acked 1636 Write 4 New Done Done true 638 Acked 1637 Read 9 New Done Done true 639 Acked 1638 Write 1 New Done Done true 640 Acked 1639 Read 5 New Done Done true 641 Acked 1640 Read 8 New Done Done true 642 Acked 1641 Read 8 New Done Done true 643 Acked 1642 Read 3 New Done Done true 644 Acked 1643 Write 1 New Done Done true 645 Acked 1644 Flush 0 New Done Done true 646 Acked 1645 Read 2 New Done Done true 647 Acked 1646 Read 1 New Done Done true 648 Acked 1647 Read 3 New Done Done true 649 Acked 1648 Read 2 New Done Done true 650 Acked 1649 Write 2 New Done Done true 651 Acked 1650 Write 10 New Done Done true 652 Acked 1651 Write 6 New Done Done true 653 Acked 1652 Read 10 New Done Done true 654 Acked 1653 Write 9 New Done Done true 655 Acked 1654 Write 6 New Done Done true 656 Acked 1655 Read 1 New Done Done true 657 Acked 1656 Write 8 New Done Done true 658 Acked 1657 Write 1 New Done Done true 659 Acked 1658 Write 7 New Done Done true 660 Acked 1659 Write 9 New Done Done true 661 Acked 1660 Read 9 New Done Done true 662 Acked 1661 Write 3 New Done Done true 663 Acked 1662 Read 5 New Done Done true 664 Acked 1663 Read 9 New Done Done true 665 Acked 1664 Write 2 New Done Done true 666 Acked 1665 Read 10 New Done Done true 667 Acked 1666 Write 10 New Done Done true 668 Acked 1667 Write 4 New Done Done true 669 Acked 1668 Read 9 New Done Done true 670 Acked 1669 Write 10 New Done Done true 671 Acked 1670 Read 2 New Done Done true 672 Acked 1671 Write 10 New Done Done true 673 Acked 1672 Read 7 New Done Done true 674 Acked 1673 Write 8 New Done Done true 675 Acked 1674 Read 6 New Done Done true 676 Acked 1675 Read 6 New Done Done true 677 Acked 1676 Flush 0 New Done Done true 678 Acked 1677 Write 9 New Done Done true 679 Acked 1678 Flush 0 New Done Done true 680 Acked 1679 Write 4 New Done Done true 681 Acked 1680 Write 3 New Done Done true 682 Acked 1681 Read 9 New Done Done true 683 Acked 1682 Read 2 New Done Done true 684 Acked 1683 Read 8 New Done Done true 685 Acked 1684 Read 5 New Done Done true 686 Acked 1685 Flush 0 New Done Done true 687 Acked 1686 Write 7 New Done Done true 688 Acked 1687 Read 6 New Done Done true 689 Acked 1688 Write 4 New Done Done true 690 Acked 1689 Write 9 New Done Done true 691 Acked 1690 Read 8 New Done Done true 692 Acked 1691 Write 9 New Done Done true 693 Acked 1692 Read 4 New Done Done true 694 Acked 1693 Read 10 New Done Done true 695 Acked 1694 Read 7 New Done Done true 696 Acked 1695 Read 2 New Done Done true 697 Acked 1696 Write 3 New Done Done true 698 Acked 1697 Read 8 New Done Done true 699 Acked 1698 Write 7 New Done Done true 700 Acked 1699 Write 7 New Done Done true 701 Acked 1700 Read 9 New Done Done true 702 Acked 1701 Read 3 New Done Done true 703 Acked 1702 Read 8 New Done Done true 704 Acked 1703 Write 10 New Done Done true 705 Acked 1704 Read 2 New Done Done true 706 Acked 1705 Write 1 New Done Done true 707 Acked 1706 Write 2 New Done Done true 708 Acked 1707 Read 8 New Done Done true 709 Acked 1708 Write 9 New Done Done true 710 Acked 1709 Read 5 New Done Done true 711 Acked 1710 Read 10 New Done Done true 712 Acked 1711 Read 5 New Done Done true 713 Acked 1712 Read 8 New Done Done true 714 Acked 1713 Write 6 New Done Done true 715 Acked 1714 Write 2 New Done Done true 716 Acked 1715 Read 10 New Done Done true 717 Acked 1716 Write 3 New Done Done true 718 Acked 1717 Read 8 New Done Done true 719 Acked 1718 Read 1 New Done Done true 720 Acked 1719 Read 5 New Done Done true 721 Acked 1720 Flush 0 New Done Done true 722 Acked 1721 Read 8 New Done Done true 723 Acked 1722 Read 8 New Done Done true 724 Acked 1723 Flush 0 New Done Done true 725 Acked 1724 Write 5 New Done Done true 726 Acked 1725 Flush 0 New Done Done true 727 Acked 1726 Read 4 New Done Done true 728 Acked 1727 Read 6 New Done Done true 729 Acked 1728 Read 3 New Done Done true 730 Acked 1729 Write 9 New Done Done true 731 Acked 1730 Read 5 New Done Done true 732 Acked 1731 Read 9 New Done Done true 733 Acked 1732 Read 1 New Done Done true 734 Acked 1733 Read 9 New Done Done true 735 Acked 1734 Write 1 New Done Done true 736 Acked 1735 Flush 0 New Done Done true 737 Acked 1736 Read 1 New Done Done true 738 Acked 1737 Write 7 New Done Done true 739 Acked 1738 Read 4 New Done Done true 740 Acked 1739 Read 4 New Done Done true 741 Acked 1740 Write 5 New Done Done true 742 Acked 1741 Write 4 New Done Done true 743 Acked 1742 Write 3 New Done Done true 744 Acked 1743 Read 4 New Done Done true 745 Acked 1744 Write 3 New Done Done true 746 Acked 1745 Write 5 New Done Done true 747 Acked 1746 Write 2 New Done Done true 748 Acked 1747 Write 7 New Done Done true 749 Acked 1748 Write 9 New Done Done true 750 Acked 1749 Read 4 New Done Done true 751 Acked 1750 Flush 0 New Done Done true 752 Acked 1751 Flush 0 New Done Done true 753 Acked 1752 Write 9 New Done Done true 754 Acked 1753 Write 2 New Done Done true 755 Acked 1754 Read 2 New Done Done true 756 Acked 1755 Read 6 New Done Done true 757 Acked 1756 Read 4 New Done Done true 758 Acked 1757 Read 3 New Done Done true 759 Acked 1758 Read 8 New Done Done true 760 Acked 1759 Write 4 New Done Done true 761 Acked 1760 Write 8 New Done Done true 762 Acked 1761 Read 7 New Done Done true 763 Acked 1762 Read 5 New Done Done true 764 Acked 1763 Write 10 New Done Done true 765 Acked 1764 Read 9 New Done Done true 766 Acked 1765 Read 1 New Done Done true 767 Acked 1766 Read 4 New Done Done true 768 Acked 1767 Write 10 New Done Done true 769 Acked 1768 Write 7 New Done Done true 770 Acked 1769 Read 3 New Done Done true 771 Acked 1770 Read 2 New Done Done true 772 Acked 1771 Write 10 New Done Done true 773 Acked 1772 Write 4 New Done Done true 774 Acked 1773 Read 8 New Done Done true 775 Acked 1774 Read 7 New Done Done true 776 Acked 1775 Read 4 New Done Done true 777 Acked 1776 Read 7 New Done Done true 778 Acked 1777 Read 9 New Done Done true 779 Acked 1778 Write 7 New Done Done true 780 Acked 1779 Write 2 New Done Done true 781 Acked 1780 Read 5 New Done Done true 782 Acked 1781 Read 3 New Done Done true 783 Acked 1782 Read 1 New Done Done true 784 Acked 1783 Write 6 New Done Done true 785 Acked 1784 Write 7 New Done Done true 786 Acked 1785 Read 7 New Done Done true 787 Acked 1786 Write 9 New Done Done true 788 Acked 1787 Write 6 New Done Done true 789 Acked 1788 Read 1 New Done Done true 790 Acked 1789 Flush 0 New Done Done true 791 Acked 1790 Read 6 New Done Done true 792 Acked 1791 Read 10 New Done Done true 793 Acked 1792 Flush 0 New Done Done true 794 Acked 1793 Read 1 New Done Done true 795 Acked 1794 Read 3 New Done Done true 796 Acked 1795 Write 2 New Done Done true 797 Acked 1796 Write 5 New Done Done true 798 Acked 1797 Read 8 New Done Done true 799 Acked 1798 Write 10 New Done Done true 800 Acked 1799 Read 9 New Done Done true 801 Acked 1800 Read 8 New Done Done true 802 Acked 1801 Read 5 New Done Done true 803 Acked 1802 Read 7 New Done Done true 804 Acked 1803 Write 4 New Done Done true 805 Acked 1804 Read 10 New Done Done true 806 Acked 1805 Write 10 New Done Done true 807 Acked 1806 Write 5 New Done Done true 808 Acked 1807 Read 8 New Done Done true 809 Acked 1808 Write 1 New Done Done true 810 Acked 1809 Write 7 New Done Done true 811 Acked 1810 Flush 0 New Done Done true 812 Acked 1811 Read 8 New Done Done true 813 Acked 1812 Flush 0 New Done Done true 814 Acked 1813 Flush 0 New Done Done true 815 Acked 1814 Flush 0 New Done Done true 816 Acked 1815 Flush 0 New Done Done true 817 Acked 1816 Write 10 New Done Done true 818 Acked 1817 Write 2 New Done Done true 819 Acked 1818 Write 5 New Done Done true 820 Acked 1819 Write 8 New Done Done true 821 Acked 1820 Read 8 New Done Done true 822 Acked 1821 Read 7 New Done Done true 823 Acked 1822 Read 6 New Done Done true 824 Acked 1823 Write 1 New Done Done true 825 Acked 1824 Write 2 New Done Done true 826 Acked 1825 Read 3 New Done Done true 827 Acked 1826 Flush 0 New Done Done true 828 Acked 1827 Write 4 New Done Done true 829 Acked 1828 Write 5 New Done Done true 830 Acked 1829 Write 9 New Done Done true 831 Acked 1830 Read 2 New Done Done true 832 Acked 1831 Write 5 New Done Done true 833 Acked 1832 Read 8 New Done Done true 834 Acked 1833 Write 1 New Done Done true 835 Acked 1834 Read 1 New Done Done true 836 Acked 1835 Write 4 New Done Done true 837 Acked 1836 Write 4 New Done Done true 838 Acked 1837 Write 2 New Done Done true 839 Acked 1838 Read 2 New Done Done true 840 Acked 1839 Read 4 New Done Done true 841 Acked 1840 Write 8 New Done Done true 842 Acked 1841 Write 4 New Done Done true 843 Acked 1842 Read 10 New Done Done true 844 Acked 1843 Write 5 New Done Done true 845 Acked 1844 Write 2 New Done Done true 846 Acked 1845 Read 10 New Done Done true 847 Acked 1846 Write 3 New Done Done true 848 Acked 1847 Read 1 New Done Done true 849 Acked 1848 Read 9 New Done Done true 850 Acked 1849 Write 1 New Done Done true 851 Acked 1850 Write 6 New Done Done true 852 Acked 1851 Flush 0 New Done Done true 853 Acked 1852 Read 1 New Done Done true 854 Acked 1853 Write 9 New Done Done true 855 Acked 1854 Read 6 New Done Done true 856 Acked 1855 Read 2 New Done Done true 857 Acked 1856 Read 8 New Done Done true 858 Acked 1857 Write 10 New Done Done true 859 Acked 1858 Flush 0 New Done Done true 860 Acked 1859 Write 10 New Done Done true 861 Acked 1860 Write 10 New Done Done true 862 Acked 1861 Read 1 New Done Done true 863 Acked 1862 Read 2 New Done Done true 864 Acked 1863 Read 8 New Done Done true 865 Acked 1864 Write 5 New Done Done true 866 Acked 1865 Read 9 New Done Done true 867 Acked 1866 Read 4 New Done Done true 868 Acked 1867 Read 3 New Done Done true 869 Acked 1868 Write 8 New Done Done true 870 Acked 1869 Read 6 New Done Done true 871 Acked 1870 Flush 0 New Done Done true 872 Acked 1871 Write 10 New Done Done true 873 Acked 1872 Read 1 New Done Done true 874 Acked 1873 Read 5 New Done Done true 875 Acked 1874 Write 10 New Done Done true 876 Acked 1875 Read 9 New Done Done true 877 Acked 1876 Read 4 New Done Done true 878 Acked 1877 Flush 0 New Done Done true 879 Acked 1878 Write 8 New Done Done true 880 Acked 1879 Read 9 New Done Done true 881 Acked 1880 Write 6 New Done Done true 882 Acked 1881 Write 6 New Done Done true 883 Acked 1882 Read 1 New Done Done true 884 Acked 1883 Write 10 New Done Done true 885 Acked 1884 Write 2 New Done Done true 886 Acked 1885 Read 1 New Done Done true 887 Acked 1886 Read 2 New Done Done true 888 Acked 1887 Write 10 New Done Done true 889 Acked 1888 Write 2 New Done Done true 890 Acked 1889 Write 2 New Done Done true 891 Acked 1890 Read 6 New Done Done true 892 Acked 1891 Read 4 New Done Done true 893 Acked 1892 Write 7 New Done Done true 894 Acked 1893 Flush 0 New Done Done true 895 Acked 1894 Write 7 New Done Done true 896 Acked 1895 Read 6 New Done Done true 897 Acked 1896 Read 2 New Done Done true 898 Acked 1897 Read 9 New Done Done true 899 Acked 1898 Read 8 New Done Done true 900 Acked 1899 Flush 0 New Done Done true 901 Acked 1900 Read 3 New Done Done true 902 Acked 1901 Write 1 New Done Done true 903 Acked 1902 Flush 0 New Done Done true 904 Acked 1903 Read 2 New Done Done true 905 Acked 1904 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1602 1904 1904 Downstairs last five completed: 1602 1601 1600 1599 1598 Upstairs last five completed: 905 904 903 902 901 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.119254637Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.11942986Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.12143573Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.12149829Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.121545673Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 1602","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.121822447Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.121874898Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.121916218Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.121956269Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:01.121995685Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1904 1903 1902 1901 1900 Upstairs last five completed: 905 904 903 902 901 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(2) stop 2 Jan 02 22:31:05.065 INFO request completed, latency_us: 443, response_code: 204, uri: /stop/cid/2, method: POST, [8830] Got stop action so:false kr:false req_id: c66b08d7-19f1-43ca-8364-f1f6b3fb5dcc, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:31:05.066 INFO request completed, latency_us: 297, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0bf3adc3-888c-4cd8-bab9-ac5719d5fc72, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:05.067212022Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:05.067301917Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:05.067333786Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:05.067355074Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:05.06738213Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:05.067504585Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:06.068386871Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:06.06845237Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:06.068480603Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:06.068507156Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:06.06852766Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:06.068579964Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:06.068690458Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:07.070252276Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:07.070318404Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:07.070346053Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:07.070370221Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:07.070396572Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:07.07041728Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:07.070458466Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:07.070557723Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:08.071680001Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:08.071744762Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:08.07177296Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:08.071800089Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:08.071828165Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:08.071873903Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:08.071973081Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:31:09.069 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1713a9ef-1848-4317-ab0c-5b560c066211, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 777 len 32768 data: 2 2 2 2 2 2 2 2 002/300 Write block 375 len 40960 data: 3 3 3 3 3 3 3 3 2 2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:09.072394051Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:09.072488075Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:09.072535689Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:09.072582251Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:09.072617122Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 2 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:09.072655763Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:09.072695316Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Write block 4794 len 32768 data: 2 2 3 4 4 3 3 3 004/300 Read block 1143 len 4096 005/300 Write block 3575 len 20480 data: 2 2 2 2 2 006/300 Read block 1962 len 4096 007/300 Read block 3826 len 4096 008/300 Write block 2648 len 32768 data: 2 2 2 3 3 3 3 3 009/300 Read block 2005 len 8192 010/300 Read block 1194 len 4096 011/300 Flush 012/300 Read block 4270 len 20480 013/300 Read block 4033 len 12288 014/300 Read block 1401 len 8192 015/300 Read block 1980 len 12288 016/300 Read block 672 len 16384 017/300 Read block 4902 len 16384 018/300 Read block 2462 len 24576 019/300 Read block 4391 len 16384 020/300 Read block 1456 len 16384 021/300 Write block 1333 len 40960 data: 2 2 2 2 2 2 2 2 2 2 022/300 Flush 023/300 Read block 4774 len 32768 024/300 Flush 025/300 Write block 373 len 8192 data: 3 3 026/300 Read block 4249 len 40960 027/300 Write block 2529 len 32768 data: 2 2 2 2 2 2 2 2 028/300 Read block 663 len 24576 029/300 Write block 2565 len 16384 data: 2 2 2 2 030/300 Write block 4023 len 20480 data: 2 2 2 2 2 031/300 Read block 3290 len 8192 032/300 Read block 960 len 4096 033/300 Read block 4913 len 4096 034/300 Read block 4577 len 36864 035/300 Read block 2023 len 4096 036/300 Write block 4266 len 20480 data: 2 2 2 2 2 037/300 Read block 1202 len 36864 038/300 Write block 3340 len 12288 data: 3 3 2 039/300 Read block 3726 len 16384 040/300 Flush 041/300 Read block 797 len 32768 042/300 Write block 3082 len 20480 data: 3 3 2 2 2 043/300 Read block 1204 len 40960 044/300 Read block 1576 len 40960 045/300 Write block 4826 len 8192 data: 2 2 046/300 Write block 1341 len 40960 data: 3 3 2 3 3 3 3 3 2 3 047/300 Write block 4631 len 40960 data: 2 2 3 3 3 3 4 3 3 3 048/300 Read block 2782 len 36864 049/300 Write block 1655 len 12288 data: 4 4 4 050/300 Read block 2445 len 20480 051/300 Read block 50 len 20480 052/300 Read block 4099 len 40960 053/300 Write block 1156 len 8192 data: 2 2 054/300 Write block 535 len 16384 data: 2 2 2 2 055/300 Write block 218 len 28672 data: 2 2 2 2 3 3 3 056/300 Write block 3990 len 36864 data: 2 2 2 3 4 4 4 4 3 057/300 Write block 1625 len 4096 data: 3 058/300 Write block 437 len 36864 data: 3 3 3 3 2 2 2 2 2 059/300 Read block 4464 len 16384 060/300 Read block 2825 len 12288 061/300 Flush 062/300 Read block 3569 len 8192 063/300 Write block 934 len 24576 data: 2 2 2 3 3 3 064/300 Read block 1182 len 32768 065/300 Read block 3848 len 4096 066/300 Write block 1648 len 24576 data: 2 2 2 2 2 2 067/300 Read block 3965 len 4096 068/300 Write block 946 len 24576 data: 3 2 2 3 4 4 069/300 Write block 3262 len 32768 data: 3 3 3 3 3 3 3 2 070/300 Read block 915 len 36864 071/300 Write block 341 len 8192 data: 2 2 072/300 Write block 1158 len 40960 data: 2 2 2 2 2 2 2 2 3 3 073/300 Write block 2752 len 28672 data: 2 2 2 2 2 2 2 074/300 Read block 1491 len 36864 075/300 Write block 3895 len 16384 data: 2 2 3 3 076/300 Read block 4475 len 4096 077/300 Write block 971 len 28672 data: 3 4 4 4 4 3 3 078/300 Read block 2622 len 28672 079/300 Flush 080/300 Write block 2061 len 8192 data: 2 2 081/300 Read block 2840 len 28672 082/300 Write block 1768 len 36864 data: 2 2 2 2 3 3 3 3 3 083/300 Write block 4534 len 36864 data: 2 2 2 2 2 2 2 2 2 084/300 Read block 4479 len 20480 085/300 Write block 4042 len 16384 data: 3 3 2 3 086/300 Write block 56 len 36864 data: 3 2 2 2 2 2 2 2 2 087/300 Read block 1999 len 20480 088/300 Write block 4534 len 12288 data: 3 3 3 089/300 Write block 4869 len 32768 data: 3 2 2 2 2 2 2 2 090/300 Read block 1704 len 36864 091/300 Read block 4171 len 36864 092/300 Write block 3018 len 20480 data: 2 2 2 2 2 093/300 Read block 2381 len 20480 094/300 Write block 3220 len 28672 data: 2 2 2 2 3 3 3 095/300 Flush 096/300 Write block 2770 len 16384 data: 6 6 6 5 097/300 Read block 831 len 8192 098/300 Read block 4740 len 20480 099/300 Flush 100/300 Read block 912 len 16384 101/300 Flush 102/300 Read block 4132 len 28672 103/300 Write block 292 len 4096 data: 2 104/300 Flush 105/300 Read block 1997 len 40960 106/300 Read block 3845 len 36864 107/300 Write block 1362 len 12288 data: 2 2 2 108/300 Write block 4145 len 8192 data: 3 3 109/300 Write block 3149 len 36864 data: 3 3 3 3 3 3 3 2 2 110/300 Read block 1017 len 32768 111/300 Flush 112/300 Write block 233 len 8192 data: 3 3 113/300 Read block 2461 len 16384 114/300 Read block 4557 len 20480 115/300 Read block 1408 len 16384 116/300 Flush 117/300 Read block 2351 len 24576 118/300 Write block 152 len 36864 data: 2 2 2 2 2 2 2 2 3 119/300 Write block 3620 len 24576 data: 3 3 4 4 4 3 120/300 Write block 1694 len 24576 data: 2 2 2 2 2 2 121/300 Read block 879 len 24576 122/300 Write block 4595 len 12288 data: 2 2 2 123/300 Write block 2031 len 12288 data: 3 3 3 124/300 Flush 125/300 Flush 126/300 Flush 127/300 Read block 3374 len 32768 128/300 Read block 4600 len 36864 129/300 Read block 3187 len 40960 130/300 Read block 2620 len 24576 131/300 Write block 3709 len 16384 data: 2 2 2 2 132/300 Write block 911 len 20480 data: 2 2 2 2 2 133/300 Read block 635 len 28672 134/300 Read block 4839 len 40960 135/300 Read block 748 len 36864 136/300 Write block 1134 len 8192 data: 4 4 137/300 Read block 313 len 20480 138/300 Write block 3358 len 24576 data: 2 3 3 3 3 3 139/300 Write block 896 len 12288 data: 3 4 4 140/300 Write block 3343 len 32768 data: 2 2 2 2 2 2 2 2 141/300 Write block 1518 len 20480 data: 2 2 2 2 2 142/300 Write block 614 len 8192 data: 2 2 143/300 Read block 1338 len 8192 144/300 Read block 3225 len 16384 145/300 Read block 3928 len 28672 146/300 Read block 385 len 20480 147/300 Write block 2307 len 32768 data: 2 2 2 2 2 3 3 3 148/300 Write block 2458 len 36864 data: 2 2 2 2 2 2 2 2 2 149/300 Read block 1202 len 24576 150/300 Write block 2942 len 40960 data: 2 2 2 2 2 2 2 2 2 2 151/300 Write block 125 len 28672 data: 2 2 2 2 3 2 2 152/300 Read block 2792 len 28672 153/300 Flush 154/300 Write block 1347 len 28672 data: 4 4 3 4 3 3 3 155/300 Write block 1040 len 4096 data: 3 156/300 Read block 467 len 36864 157/300 Read block 4965 len 36864 158/300 Write block 2150 len 24576 data: 2 2 2 2 2 2 159/300 Read block 808 len 28672 160/300 Write block 1365 len 8192 data: 2 2 161/300 Read block 1451 len 32768 162/300 Write block 310 len 16384 data: 3 3 3 2 163/300 Read block 1749 len 20480 164/300 Read block 4056 len 36864 165/300 Read block 1385 len 4096 166/300 Read block 3943 len 4096 167/300 Read block 192 len 36864 168/300 Write block 663 len 36864 data: 3 3 3 3 4 2 2 2 2 169/300 Write block 1183 len 8192 data: 2 2 170/300 Write block 2913 len 20480 data: 2 2 2 2 2 171/300 Read block 4124 len 20480 172/300 Read block 4330 len 12288 173/300 Write block 2716 len 4096 data: 3 174/300 Read block 4292 len 16384 175/300 Flush 176/300 Read block 3173 len 36864 177/300 Read block 3797 len 20480 178/300 Flush 179/300 Read block 808 len 16384 180/300 Read block 2732 len 8192 181/300 Write block 1700 len 12288 data: 2 2 2 182/300 Write block 3366 len 24576 data: 3 3 3 2 2 2 183/300 Flush 184/300 Flush 185/300 Read block 3693 len 36864 186/300 Flush 187/300 Read block 114 len 28672 188/300 Write block 4964 len 8192 data: 2 2 189/300 Write block 883 len 40960 data: 2 2 2 2 2 2 2 3 3 4 190/300 Read block 1805 len 4096 191/300 Write block 3384 len 4096 data: 2 192/300 Write block 1102 len 16384 data: 2 2 2 3 193/300 Read block 3799 len 16384 194/300 Read block 4719 len 36864 195/300 Write block 632 len 16384 data: 2 2 2 2 196/300 Read block 641 len 20480 197/300 Read block 4250 len 24576 198/300 Write block 3837 len 20480 data: 3 3 2 2 2 199/300 Write block 1969 len 24576 data: 2 2 2 2 2 2 200/300 Read block 4210 len 4096 201/300 Read block 1317 len 12288 202/300 Write block 571 len 8192 data: 2 2 203/300 Write block 4073 len 16384 data: 2 2 2 2 204/300 Read block 1959 len 36864 205/300 Write block 793 len 20480 data: 2 2 2 2 2 206/300 Flush 207/300 Write block 4605 len 28672 data: 4 4 3 2 3 3 3 208/300 Read block 4871 len 8192 209/300 Read block 3185 len 32768 210/300 Write block 3039 len 40960 data: 2 2 2 2 2 2 2 2 2 2 211/300 Flush 212/300 Flush 213/300 Write block 2139 len 40960 data: 2 2 2 2 2 2 2 2 2 2 214/300 Write block 4409 len 8192 data: 2 2 215/300 Write block 534 len 40960 data: 2 3 3 3 3 2 2 2 2 2 216/300 Write block 1094 len 12288 data: 2 2 2 217/300 Read block 2591 len 40960 218/300 Read block 4917 len 4096 219/300 Write block 2917 len 32768 data: 3 2 2 2 2 3 2 2 220/300 Read block 3399 len 32768 221/300 Read block 2203 len 24576 222/300 Read block 3443 len 32768 223/300 Write block 4225 len 40960 data: 2 2 2 2 2 2 2 2 2 2 224/300 Read block 733 len 28672 225/300 Write block 4186 len 16384 data: 2 3 3 3 226/300 Write block 4709 len 8192 data: 2 3 227/300 Read block 3345 len 20480 228/300 Write block 318 len 8192 data: 2 2 229/300 Read block 813 len 12288 230/300 Flush 231/300 Flush 232/300 Write block 2945 len 40960 data: 3 3 3 3 3 3 3 2 2 2 233/300 Write block 245 len 16384 data: 2 2 2 2 234/300 Read block 3881 len 40960 235/300 Read block 1233 len 40960 236/300 Flush 237/300 Read block 3797 len 28672 238/300 Read block 770 len 20480 239/300 Write block 4268 len 20480 data: 3 3 3 2 2 240/300 Write block 2710 len 32768 data: 2 2 2 2 3 3 4 3 241/300 Write block 4011 len 32768 data: 2 2 2 2 2 2 2 2 242/300 Flush 243/300 Flush 244/300 Read block 3868 len 8192 245/300 Read block 2975 len 40960 246/300 Read block 4167 len 16384 247/300 Write block 3763 len 40960 data: 2 2 2 2 2 2 2 2 2 2 248/300 Write block 4944 len 8192 data: 2 2 249/300 Read block 1243 len 24576 250/300 Write block 416 len 40960 data: 2 2 2 2 2 2 2 2 2 2 251/300 Read block 4559 len 20480 252/300 Write block 3120 len 32768 data: 2 2 2 2 2 2 2 2 253/300 Read block 1201 len 16384 254/300 Read block 244 len 4096 255/300 Write block 3235 len 36864 data: 2 2 3 3 3 4 4 3 4 256/300 Read block 4777 len 4096 257/300 Read block 2999 len 40960 258/300 Read block 4962 len 28672 259/300 Read block 2013 len 24576 260/300 Read block 1713 len 40960 261/300 Write block 3868 len 28672 data: 3 3 3 3 3 2 2 262/300 Flush 263/300 Flush 264/300 Read block 588 len 12288 265/300 Flush 266/300 Read block 2651 len 16384 267/300 Write block 643 len 16384 data: 3 3 3 3 268/300 Write block 3739 len 40960 data: 3 3 2 2 2 2 2 2 2 2 269/300 Write block 4516 len 28672 data: 3 3 3 3 3 3 3 270/300 Flush 271/300 Write block 200 len 16384 data: 2 2 2 2 272/300 Write block 1810 len 8192 data: 2 2 273/300 Read block 3456 len 12288 274/300 Read block 3220 len 40960 275/300 Read block 176 len 20480 276/300 Write block 3136 len 4096 data: 4 277/300 Read block 1092 len 20480 278/300 Read block 4071 len 40960 279/300 Flush 280/300 Read block 2874 len 36864 281/300 Read block 2941 len 20480 282/300 Write block 144 len 28672 data: 2 2 2 2 2 2 2 283/300 Read block 125 len 4096 284/300 Write block 1526 len 8192 data: 2 2 285/300 Write block 985 len 4096 data: 2 286/300 Flush 287/300 Read block 3466 len 4096 288/300 Write block 3914 len 28672 data: 3 3 2 2 2 2 2 289/300 Flush 290/300 Flush 291/300 Read block 4279 len 4096 292/300 Write block 4829 len 20480 data: 2 2 2 2 2 293/300 Write block 15 len 24576 data: 4 4 4 4 3 2 294/300 Read block 1366 len 28672 295/300 Write block 813 len 28672 data: 3 3 3 3 3 3 2 296/300 Read block 3679 len 4096 297/300 Read block 4036 len 24576 298/300 Read block 4542 len 4096 299/300 Read block 4560 len 8192 300/300 Read block 1764 len 16384 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:31:09.926 INFO request completed, latency_us: 305, response_code: 204, uri: /start/cid/2, method: POST, req_id: b1e9423a-fea2-466c-83b4-5d5cc59e13f4, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 906 Acked 1905 Write 8 Done Done New true 907 Acked 1906 Write 10 Done Done New true 908 Acked 1907 Write 8 Done Done New false 909 Acked 1908 Read 1 Done Done New false 910 Acked 1909 Write 5 Done Done New false 911 Acked 1910 Read 1 Done Done New false 912 Acked 1911 Read 1 Done Done New false 913 Acked 1912 Write 8 Done Done New false 914 Acked 1913 Read 2 Done Done New false 915 Acked 1914 Read 1 Done Done New false 916 Acked 1915 Flush 0 Done Done New false 917 Acked 1916 Read 5 Done Done New false 918 Acked 1917 Read 3 Done Done New false 919 Acked 1918 Read 2 Done Done New false 920 Acked 1919 Read 3 Done Done New false 921 Acked 1920 Read 4 Done Done New false 922 Acked 1921 Read 4 Done Done New false 923 Acked 1922 Read 6 Done Done New false 924 Acked 1923 Read 4 Done Done New false 925 Acked 1924 Read 4 Done Done New false 926 Acked 1925 Write 10 Done Done New false 927 Acked 1926 Flush 0 Done Done New false 928 Acked 1927 Read 8 Done Done New false 929 Acked 1928 Flush 0 Done Done New false 930 Acked 1929 Write 2 Done Done New false 931 Acked 1930 Read 10 Done Done New false 932 Acked 1931 Write 8 Done Done New false 933 Acked 1932 Read 6 Done Done New false 934 Acked 1933 Write 4 Done Done New false 935 Acked 1934 Write 5 Done Done New false 936 Acked 1935 Read 2 Done Done New false 937 Acked 1936 Read 1 Done Done New false 938 Acked 1937 Read 1 Done Done New false 939 Acked 1938 Read 9 Done Done New false 940 Acked 1939 Read 1 Done Done New false 941 Acked 1940 Write 5 Done Done New false 942 Acked 1941 Read 9 Done Done New false 943 Acked 1942 Write 3 Done Done New false 944 Acked 1943 Read 4 Done Done New false 945 Acked 1944 Flush 0 Done Done New false 946 Acked 1945 Read 8 Done Done New false 947 Acked 1946 Write 5 Done Done New false 948 Acked 1947 Read 10 Done Done New false 949 Acked 1948 Read 10 Done Done New false 950 Acked 1949 Write 2 Done Done New false 951 Acked 1950 Write 10 Done Done New false 952 Acked 1951 Write 10 Done Done New false 953 Acked 1952 Read 9 Done Done New false 954 Acked 1953 Write 3 Done Done New false 955 Acked 1954 Read 5 Done Done New false 956 Acked 1955 Read 5 Done Done New false 957 Acked 1956 Read 10 Done Done New false 958 Acked 1957 Write 2 Done Done New false 959 Acked 1958 Write 4 Done Done New false 960 Acked 1959 Write 7 Done Done New false 961 Acked 1960 Write 9 Done Done New false 962 Acked 1961 Write 1 Done Done New false 963 Acked 1962 Write 9 Done Done New false 964 Acked 1963 Read 4 Done Done New false 965 Acked 1964 Read 3 Done Done New false 966 Acked 1965 Flush 0 Done Done New false 967 Acked 1966 Read 2 Done Done New false 968 Acked 1967 Write 6 Done Done New false 969 Acked 1968 Read 8 Done Done New false 970 Acked 1969 Read 1 Done Done New false 971 Acked 1970 Write 6 Done Done New false 972 Acked 1971 Read 1 Done Done New false 973 Acked 1972 Write 6 Done Done New false 974 Acked 1973 Write 8 Done Done New false 975 Acked 1974 Read 9 Done Done New false 976 Acked 1975 Write 2 Done Done New false 977 Acked 1976 Write 10 Done Done New false 978 Acked 1977 Write 7 Done Done New false 979 Acked 1978 Read 9 Done Done New false 980 Acked 1979 Flush 0 Done Done New false 981 Acked 1980 Write 4 Done Done New false 982 Acked 1981 Read 1 Done Done New false 983 Acked 1982 Write 7 Done Done New false 984 Acked 1983 Read 7 Done Done New false 985 Acked 1984 Flush 0 Done Done New false 986 Acked 1985 Write 2 Done Done New false 987 Acked 1986 Read 7 Done Done New false 988 Acked 1987 Write 9 Done Done New false 989 Acked 1988 Write 9 Done Done New false 990 Acked 1989 Read 5 Done Done New false 991 Acked 1990 Write 4 Done Done New false 992 Acked 1991 Write 9 Done Done New false 993 Acked 1992 Read 5 Done Done New false 994 Acked 1993 Write 3 Done Done New false 995 Acked 1994 Write 8 Done Done New false 996 Acked 1995 Read 9 Done Done New false 997 Acked 1996 Read 9 Done Done New false 998 Acked 1997 Write 5 Done Done New false 999 Acked 1998 Read 5 Done Done New false 1000 Acked 1999 Write 7 Done Done New false 1001 Acked 2000 Flush 0 Done Done New false 1002 Acked 2001 Write 4 Done Done New false 1003 Acked 2002 Read 2 Done Done New false 1004 Acked 2003 Read 5 Done Done New false 1005 Acked 2004 Flush 0 Done Done New false 1006 Acked 2005 Read 4 Done Done New false 1007 Acked 2006 Flush 0 Done Done New false 1008 Acked 2007 Read 7 Done Done New false 1009 Acked 2008 Write 1 Done Done New false 1010 Acked 2009 Flush 0 Done Done New false 1011 Acked 2010 Read 10 Done Done New false 1012 Acked 2011 Read 9 Done Done New false 1013 Acked 2012 Write 3 Done Done New false 1014 Acked 2013 Write 2 Done Done New false 1015 Acked 2014 Write 9 Done Done New false 1016 Acked 2015 Read 8 Done Done New false 1017 Acked 2016 Flush 0 Done Done New false 1018 Acked 2017 Write 2 Done Done New false 1019 Acked 2018 Read 4 Done Done New false 1020 Acked 2019 Read 5 Done Done New false 1021 Acked 2020 Read 4 Done Done New false 1022 Acked 2021 Flush 0 Done Done New false 1023 Acked 2022 Read 6 Done Done New false 1024 Acked 2023 Write 9 Done Done New false 1025 Acked 2024 Write 6 Done Done New false 1026 Acked 2025 Write 6 Done Done New false 1027 Acked 2026 Read 6 Done Done New false 1028 Acked 2027 Write 3 Done Done New false 1029 Acked 2028 Write 3 Done Done New false 1030 Acked 2029 Flush 0 Done Done New false 1031 Acked 2030 Flush 0 Done Done New false 1032 Acked 2031 Flush 0 Done Done New false 1033 Acked 2032 Read 8 Done Done New false 1034 Acked 2033 Read 9 Done Done New false 1035 Acked 2034 Read 10 Done Done New false 1036 Acked 2035 Read 6 Done Done New false 1037 Acked 2036 Write 4 Done Done New false 1038 Acked 2037 Write 5 Done Done New false 1039 Acked 2038 Read 7 Done Done New false 1040 Acked 2039 Read 10 Done Done New false 1041 Acked 2040 Read 9 Done Done New false 1042 Acked 2041 Write 2 Done Done New false 1043 Acked 2042 Read 5 Done Done New false 1044 Acked 2043 Write 6 Done Done New false 1045 Acked 2044 Write 3 Done Done New false 1046 Acked 2045 Write 8 Done Done New false 1047 Acked 2046 Write 5 Done Done New false 1048 Acked 2047 Write 2 Done Done New false 1049 Acked 2048 Read 2 Done Done New false 1050 Acked 2049 Read 4 Done Done New false 1051 Acked 2050 Read 7 Done Done New false 1052 Acked 2051 Read 5 Done Done New false 1053 Acked 2052 Write 8 Done Done New false 1054 Acked 2053 Write 9 Done Done New false 1055 Acked 2054 Read 6 Done Done New false 1056 Acked 2055 Write 10 Done Done New false 1057 Acked 2056 Write 7 Done Done New false 1058 Acked 2057 Read 7 Done Done New false 1059 Acked 2058 Flush 0 Done Done New false 1060 Acked 2059 Write 7 Done Done New false 1061 Acked 2060 Write 1 Done Done New false 1062 Acked 2061 Read 9 Done Done New false 1063 Acked 2062 Read 9 Done Done New false 1064 Acked 2063 Write 6 Done Done New false 1065 Acked 2064 Read 7 Done Done New false 1066 Acked 2065 Write 2 Done Done New false 1067 Acked 2066 Read 8 Done Done New false 1068 Acked 2067 Write 4 Done Done New false 1069 Acked 2068 Read 5 Done Done New false 1070 Acked 2069 Read 9 Done Done New false 1071 Acked 2070 Read 1 Done Done New false 1072 Acked 2071 Read 1 Done Done New false 1073 Acked 2072 Read 9 Done Done New false 1074 Acked 2073 Write 9 Done Done New false 1075 Acked 2074 Write 2 Done Done New false 1076 Acked 2075 Write 5 Done Done New false 1077 Acked 2076 Read 5 Done Done New false 1078 Acked 2077 Read 3 Done Done New false 1079 Acked 2078 Write 1 Done Done New false 1080 Acked 2079 Read 4 Done Done New false 1081 Acked 2080 Flush 0 Done Done New false 1082 Acked 2081 Read 9 Done Done New false 1083 Acked 2082 Read 5 Done Done New false 1084 Acked 2083 Flush 0 Done Done New false 1085 Acked 2084 Read 4 Done Done New false 1086 Acked 2085 Read 2 Done Done New false 1087 Acked 2086 Write 3 Done Done New false 1088 Acked 2087 Write 6 Done Done New false 1089 Acked 2088 Flush 0 Done Done New false 1090 Acked 2089 Flush 0 Done Done New false 1091 Acked 2090 Read 9 Done Done New false 1092 Acked 2091 Flush 0 Done Done New false 1093 Acked 2092 Read 7 Done Done New false 1094 Acked 2093 Write 2 Done Done New false 1095 Acked 2094 Write 10 Done Done New false 1096 Acked 2095 Read 1 Done Done New false 1097 Acked 2096 Write 1 Done Done New false 1098 Acked 2097 Write 4 Done Done New false 1099 Acked 2098 Read 4 Done Done New false 1100 Acked 2099 Read 9 Done Done New false 1101 Acked 2100 Write 4 Done Done New false 1102 Acked 2101 Read 5 Done Done New false 1103 Acked 2102 Read 6 Done Done New false 1104 Acked 2103 Write 5 Done Done New false 1105 Acked 2104 Write 6 Done Done New false 1106 Acked 2105 Read 1 Done Done New false 1107 Acked 2106 Read 3 Done Done New false 1108 Acked 2107 Write 2 Done Done New false 1109 Acked 2108 Write 4 Done Done New false 1110 Acked 2109 Read 9 Done Done New false 1111 Acked 2110 Write 5 Done Done New false 1112 Acked 2111 Flush 0 Done Done New false 1113 Acked 2112 Write 7 Done Done New false 1114 Acked 2113 Read 2 Done Done New false 1115 Acked 2114 Read 8 Done Done New false 1116 Acked 2115 Write 10 Done Done New false 1117 Acked 2116 Flush 0 Done Done New false 1118 Acked 2117 Flush 0 Done Done New false 1119 Acked 2118 Write 10 Done Done New false 1120 Acked 2119 Write 2 Done Done New false 1121 Acked 2120 Write 10 Done Done New false 1122 Acked 2121 Write 3 Done Done New false 1123 Acked 2122 Read 10 Done Done New false 1124 Acked 2123 Read 1 Done Done New false 1125 Acked 2124 Write 8 Done Done New false 1126 Acked 2125 Read 8 Done Done New false 1127 Acked 2126 Read 6 Done Done New false 1128 Acked 2127 Read 8 Done Done New false 1129 Acked 2128 Write 10 Done Done New false 1130 Acked 2129 Read 7 Done Done New false 1131 Acked 2130 Write 4 Done Done New false 1132 Acked 2131 Write 2 Done Done New false 1133 Acked 2132 Read 5 Done Done New false 1134 Acked 2133 Write 2 Done Done New false 1135 Acked 2134 Read 3 Done Done New false 1136 Acked 2135 Flush 0 Done Done New false 1137 Acked 2136 Flush 0 Done Done New false 1138 Acked 2137 Write 10 Done Done New false 1139 Acked 2138 Write 4 Done Done New false 1140 Acked 2139 Read 10 Done Done New false 1141 Acked 2140 Read 10 Done Done New false 1142 Acked 2141 Flush 0 Done Done New false 1143 Acked 2142 Read 7 Done Done New false 1144 Acked 2143 Read 5 Done Done New false 1145 Acked 2144 Write 5 Done Done New false 1146 Acked 2145 Write 8 Done Done New false 1147 Acked 2146 Write 8 Done Done New false 1148 Acked 2147 Flush 0 Done Done New false 1149 Acked 2148 Flush 0 Done Done New false 1150 Acked 2149 Read 2 Done Done New false 1151 Acked 2150 Read 10 Done Done New false 1152 Acked 2151 Read 4 Done Done New false 1153 Acked 2152 Write 10 Done Done New false 1154 Acked 2153 Write 2 Done Done New false 1155 Acked 2154 Read 6 Done Done New false 1156 Acked 2155 Write 10 Done Done New false 1157 Acked 2156 Read 5 Done Done New false 1158 Acked 2157 Write 8 Done Done New false 1159 Acked 2158 Read 4 Done Done New false 1160 Acked 2159 Read 1 Done Done New false 1161 Acked 2160 Write 9 Done Done New false 1162 Acked 2161 Read 1 Done Done New false 1163 Acked 2162 Read 10 Done Done New false 1164 Acked 2163 Read 7 Done Done New false 1165 Acked 2164 Read 6 Done Done New false 1166 Acked 2165 Read 10 Done Done New false 1167 Acked 2166 Write 7 Done Done New false 1168 Acked 2167 Flush 0 Done Done New false 1169 Acked 2168 Flush 0 Done Done New false 1170 Acked 2169 Read 3 Done Done New false 1171 Acked 2170 Flush 0 Done Done New false 1172 Acked 2171 Read 4 Done Done New false 1173 Acked 2172 Write 4 Done Done New false 1174 Acked 2173 Write 10 Done Done New false 1175 Acked 2174 Write 7 Done Done New false 1176 Acked 2175 Flush 0 Done Done New false 1177 Acked 2176 Write 4 Done Done New false 1178 Acked 2177 Write 2 Done Done New false 1179 Acked 2178 Read 3 Done Done New false 1180 Acked 2179 Read 10 Done Done New false 1181 Acked 2180 Read 5 Done Done New false 1182 Acked 2181 Write 1 Done Done New false 1183 Acked 2182 Read 5 Done Done New false 1184 Acked 2183 Read 10 Done Done New false 1185 Acked 2184 Flush 0 Done Done New false 1186 Acked 2185 Read 9 Done Done New false 1187 Acked 2186 Read 5 Done Done New false 1188 Acked 2187 Write 7 Done Done New false 1189 Acked 2188 Read 1 Done Done New false 1190 Acked 2189 Write 2 Done Done New false 1191 Acked 2190 Write 1 Done Done New false 1192 Acked 2191 Flush 0 Done Done New false 1193 Acked 2192 Read 1 Done Done New false 1194 Acked 2193 Write 7 Done Done New false 1195 Acked 2194 Flush 0 Done Done New false 1196 Acked 2195 Flush 0 Done Done New false 1197 Acked 2196 Read 1 Done Done New false 1198 Acked 2197 Write 5 Done Done New false 1199 Acked 2198 Write 6 Done Done New false 1200 Acked 2199 Read 7 Done Done New false 1201 Acked 2200 Write 7 Done Done New false 1202 Acked 2201 Read 1 Done Done New false 1203 Acked 2202 Read 6 Done Done New false 1204 Acked 2203 Read 1 Done Done New false 1205 Acked 2204 Read 2 Done Done New false 1206 Acked 2205 Read 4 Done Done New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 301 301 Sent 0 0 0 0 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2195 2195 1904 Downstairs last five completed: 1904 1903 1902 1901 1900 Upstairs last five completed: 1206 1205 1204 1203 1202 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:10.07375144Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:10.073821428Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:10.073857864Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:10.073885426Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:10.07390591Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 301 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:10.073947436Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:10.074195692Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:11.075289742Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:11.075357079Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:11.075384985Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:11.075411855Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:11.075432421Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:11.075495791Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:11.075738734Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:12.076378421Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:12.076455488Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:12.076483978Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:12.076508117Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:12.07653407Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:12.076554687Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:12.076588971Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:12.07685951Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:13.077656256Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:13.077719514Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:13.077746791Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:13.077780642Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:13.077802082Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:13.077902354Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:13.078139641Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 906 Acked 1905 Write 8 Done Done New true 907 Acked 1906 Write 10 Done Done New true 908 Acked 1907 Write 8 Done Done New true 909 Acked 1908 Read 1 Done Done New true 910 Acked 1909 Write 5 Done Done New true 911 Acked 1910 Read 1 Done Done New true 912 Acked 1911 Read 1 Done Done New true 913 Acked 1912 Write 8 Done Done New true 914 Acked 1913 Read 2 Done Done New true 915 Acked 1914 Read 1 Done Done New true 916 Acked 1915 Flush 0 Done Done New true 917 Acked 1916 Read 5 Done Done New true 918 Acked 1917 Read 3 Done Done New true 919 Acked 1918 Read 2 Done Done New true 920 Acked 1919 Read 3 Done Done New true 921 Acked 1920 Read 4 Done Done New true 922 Acked 1921 Read 4 Done Done New true 923 Acked 1922 Read 6 Done Done New true 924 Acked 1923 Read 4 Done Done New true 925 Acked 1924 Read 4 Done Done New true 926 Acked 1925 Write 10 Done Done New true 927 Acked 1926 Flush 0 Done Done New true 928 Acked 1927 Read 8 Done Done New true 929 Acked 1928 Flush 0 Done Done New true 930 Acked 1929 Write 2 Done Done New true 931 Acked 1930 Read 10 Done Done New true 932 Acked 1931 Write 8 Done Done New true 933 Acked 1932 Read 6 Done Done New true 934 Acked 1933 Write 4 Done Done New true 935 Acked 1934 Write 5 Done Done New true 936 Acked 1935 Read 2 Done Done New true 937 Acked 1936 Read 1 Done Done New true 938 Acked 1937 Read 1 Done Done New true 939 Acked 1938 Read 9 Done Done New true 940 Acked 1939 Read 1 Done Done New true 941 Acked 1940 Write 5 Done Done New true 942 Acked 1941 Read 9 Done Done New true 943 Acked 1942 Write 3 Done Done New true 944 Acked 1943 Read 4 Done Done New true 945 Acked 1944 Flush 0 Done Done New true 946 Acked 1945 Read 8 Done Done New true 947 Acked 1946 Write 5 Done Done New true 948 Acked 1947 Read 10 Done Done New true 949 Acked 1948 Read 10 Done Done New true 950 Acked 1949 Write 2 Done Done New true 951 Acked 1950 Write 10 Done Done New true 952 Acked 1951 Write 10 Done Done New true 953 Acked 1952 Read 9 Done Done New true 954 Acked 1953 Write 3 Done Done New true 955 Acked 1954 Read 5 Done Done New true 956 Acked 1955 Read 5 Done Done New true 957 Acked 1956 Read 10 Done Done New true 958 Acked 1957 Write 2 Done Done New true 959 Acked 1958 Write 4 Done Done New true 960 Acked 1959 Write 7 Done Done New true 961 Acked 1960 Write 9 Done Done New true 962 Acked 1961 Write 1 Done Done New true 963 Acked 1962 Write 9 Done Done New true 964 Acked 1963 Read 4 Done Done New true 965 Acked 1964 Read 3 Done Done New true 966 Acked 1965 Flush 0 Done Done New true 967 Acked 1966 Read 2 Done Done New true 968 Acked 1967 Write 6 Done Done New true 969 Acked 1968 Read 8 Done Done New true 970 Acked 1969 Read 1 Done Done New true 971 Acked 1970 Write 6 Done Done New true 972 Acked 1971 Read 1 Done Done New true 973 Acked 1972 Write 6 Done Done New true 974 Acked 1973 Write 8 Done Done New true 975 Acked 1974 Read 9 Done Done New true 976 Acked 1975 Write 2 Done Done New true 977 Acked 1976 Write 10 Done Done New true 978 Acked 1977 Write 7 Done Done New true 979 Acked 1978 Read 9 Done Done New true 980 Acked 1979 Flush 0 Done Done New true 981 Acked 1980 Write 4 Done Done New true 982 Acked 1981 Read 1 Done Done New true 983 Acked 1982 Write 7 Done Done New true 984 Acked 1983 Read 7 Done Done New true 985 Acked 1984 Flush 0 Done Done New true 986 Acked 1985 Write 2 Done Done New true 987 Acked 1986 Read 7 Done Done New true 988 Acked 1987 Write 9 Done Done New true 989 Acked 1988 Write 9 Done Done New true 990 Acked 1989 Read 5 Done Done New true 991 Acked 1990 Write 4 Done Done New true 992 Acked 1991 Write 9 Done Done New true 993 Acked 1992 Read 5 Done Done New true 994 Acked 1993 Write 3 Done Done New true 995 Acked 1994 Write 8 Done Done New true 996 Acked 1995 Read 9 Done Done New true 997 Acked 1996 Read 9 Done Done New true 998 Acked 1997 Write 5 Done Done New true 999 Acked 1998 Read 5 Done Done New true 1000 Acked 1999 Write 7 Done Done New true 1001 Acked 2000 Flush 0 Done Done New true 1002 Acked 2001 Write 4 Done Done New true 1003 Acked 2002 Read 2 Done Done New true 1004 Acked 2003 Read 5 Done Done New true 1005 Acked 2004 Flush 0 Done Done New true 1006 Acked 2005 Read 4 Done Done New true 1007 Acked 2006 Flush 0 Done Done New true 1008 Acked 2007 Read 7 Done Done New true 1009 Acked 2008 Write 1 Done Done New true 1010 Acked 2009 Flush 0 Done Done New true 1011 Acked 2010 Read 10 Done Done New true 1012 Acked 2011 Read 9 Done Done New true 1013 Acked 2012 Write 3 Done Done New true 1014 Acked 2013 Write 2 Done Done New true 1015 Acked 2014 Write 9 Done Done New true 1016 Acked 2015 Read 8 Done Done New true 1017 Acked 2016 Flush 0 Done Done New true 1018 Acked 2017 Write 2 Done Done New true 1019 Acked 2018 Read 4 Done Done New true 1020 Acked 2019 Read 5 Done Done New true 1021 Acked 2020 Read 4 Done Done New true 1022 Acked 2021 Flush 0 Done Done New true 1023 Acked 2022 Read 6 Done Done New true 1024 Acked 2023 Write 9 Done Done New true 1025 Acked 2024 Write 6 Done Done New true 1026 Acked 2025 Write 6 Done Done New true 1027 Acked 2026 Read 6 Done Done New true 1028 Acked 2027 Write 3 Done Done New true 1029 Acked 2028 Write 3 Done Done New true 1030 Acked 2029 Flush 0 Done Done New true 1031 Acked 2030 Flush 0 Done Done New true 1032 Acked 2031 Flush 0 Done Done New true 1033 Acked 2032 Read 8 Done Done New true 1034 Acked 2033 Read 9 Done Done New true 1035 Acked 2034 Read 10 Done Done New true 1036 Acked 2035 Read 6 Done Done New true 1037 Acked 2036 Write 4 Done Done New true 1038 Acked 2037 Write 5 Done Done New true 1039 Acked 2038 Read 7 Done Done New true 1040 Acked 2039 Read 10 Done Done New true 1041 Acked 2040 Read 9 Done Done New true 1042 Acked 2041 Write 2 Done Done New true 1043 Acked 2042 Read 5 Done Done New true 1044 Acked 2043 Write 6 Done Done New true 1045 Acked 2044 Write 3 Done Done New true 1046 Acked 2045 Write 8 Done Done New true 1047 Acked 2046 Write 5 Done Done New true 1048 Acked 2047 Write 2 Done Done New true 1049 Acked 2048 Read 2 Done Done New true 1050 Acked 2049 Read 4 Done Done New true 1051 Acked 2050 Read 7 Done Done New true 1052 Acked 2051 Read 5 Done Done New true 1053 Acked 2052 Write 8 Done Done New true 1054 Acked 2053 Write 9 Done Done New true 1055 Acked 2054 Read 6 Done Done New true 1056 Acked 2055 Write 10 Done Done New true 1057 Acked 2056 Write 7 Done Done New true 1058 Acked 2057 Read 7 Done Done New true 1059 Acked 2058 Flush 0 Done Done New true 1060 Acked 2059 Write 7 Done Done New true 1061 Acked 2060 Write 1 Done Done New true 1062 Acked 2061 Read 9 Done Done New true 1063 Acked 2062 Read 9 Done Done New true 1064 Acked 2063 Write 6 Done Done New true 1065 Acked 2064 Read 7 Done Done New true 1066 Acked 2065 Write 2 Done Done New true 1067 Acked 2066 Read 8 Done Done New true 1068 Acked 2067 Write 4 Done Done New true 1069 Acked 2068 Read 5 Done Done New true 1070 Acked 2069 Read 9 Done Done New true 1071 Acked 2070 Read 1 Done Done New true 1072 Acked 2071 Read 1 Done Done New true 1073 Acked 2072 Read 9 Done Done New true 1074 Acked 2073 Write 9 Done Done New true 1075 Acked 2074 Write 2 Done Done New true 1076 Acked 2075 Write 5 Done Done New true 1077 Acked 2076 Read 5 Done Done New true 1078 Acked 2077 Read 3 Done Done New true 1079 Acked 2078 Write 1 Done Done New true 1080 Acked 2079 Read 4 Done Done New true 1081 Acked 2080 Flush 0 Done Done New true 1082 Acked 2081 Read 9 Done Done New true 1083 Acked 2082 Read 5 Done Done New true 1084 Acked 2083 Flush 0 Done Done New true 1085 Acked 2084 Read 4 Done Done New true 1086 Acked 2085 Read 2 Done Done New true 1087 Acked 2086 Write 3 Done Done New true 1088 Acked 2087 Write 6 Done Done New true 1089 Acked 2088 Flush 0 Done Done New true 1090 Acked 2089 Flush 0 Done Done New true 1091 Acked 2090 Read 9 Done Done New true 1092 Acked 2091 Flush 0 Done Done New true 1093 Acked 2092 Read 7 Done Done New true 1094 Acked 2093 Write 2 Done Done New true 1095 Acked 2094 Write 10 Done Done New true 1096 Acked 2095 Read 1 Done Done New true 1097 Acked 2096 Write 1 Done Done New true 1098 Acked 2097 Write 4 Done Done New true 1099 Acked 2098 Read 4 Done Done New true 1100 Acked 2099 Read 9 Done Done New true 1101 Acked 2100 Write 4 Done Done New true 1102 Acked 2101 Read 5 Done Done New true 1103 Acked 2102 Read 6 Done Done New true 1104 Acked 2103 Write 5 Done Done New true 1105 Acked 2104 Write 6 Done Done New true 1106 Acked 2105 Read 1 Done Done New true 1107 Acked 2106 Read 3 Done Done New true 1108 Acked 2107 Write 2 Done Done New true 1109 Acked 2108 Write 4 Done Done New true 1110 Acked 2109 Read 9 Done Done New true 1111 Acked 2110 Write 5 Done Done New true 1112 Acked 2111 Flush 0 Done Done New true 1113 Acked 2112 Write 7 Done Done New true 1114 Acked 2113 Read 2 Done Done New true 1115 Acked 2114 Read 8 Done Done New true 1116 Acked 2115 Write 10 Done Done New true 1117 Acked 2116 Flush 0 Done Done New true 1118 Acked 2117 Flush 0 Done Done New true 1119 Acked 2118 Write 10 Done Done New true 1120 Acked 2119 Write 2 Done Done New true 1121 Acked 2120 Write 10 Done Done New true 1122 Acked 2121 Write 3 Done Done New true 1123 Acked 2122 Read 10 Done Done New true 1124 Acked 2123 Read 1 Done Done New true 1125 Acked 2124 Write 8 Done Done New true 1126 Acked 2125 Read 8 Done Done New true 1127 Acked 2126 Read 6 Done Done New true 1128 Acked 2127 Read 8 Done Done New true 1129 Acked 2128 Write 10 Done Done New true 1130 Acked 2129 Read 7 Done Done New true 1131 Acked 2130 Write 4 Done Done New true 1132 Acked 2131 Write 2 Done Done New true 1133 Acked 2132 Read 5 Done Done New true 1134 Acked 2133 Write 2 Done Done New true 1135 Acked 2134 Read 3 Done Done New true 1136 Acked 2135 Flush 0 Done Done New true 1137 Acked 2136 Flush 0 Done Done New true 1138 Acked 2137 Write 10 Done Done New true 1139 Acked 2138 Write 4 Done Done New true 1140 Acked 2139 Read 10 Done Done New true 1141 Acked 2140 Read 10 Done Done New true 1142 Acked 2141 Flush 0 Done Done New true 1143 Acked 2142 Read 7 Done Done New true 1144 Acked 2143 Read 5 Done Done New true 1145 Acked 2144 Write 5 Done Done New true 1146 Acked 2145 Write 8 Done Done New true 1147 Acked 2146 Write 8 Done Done New true 1148 Acked 2147 Flush 0 Done Done New true 1149 Acked 2148 Flush 0 Done Done New true 1150 Acked 2149 Read 2 Done Done New true 1151 Acked 2150 Read 10 Done Done New true 1152 Acked 2151 Read 4 Done Done New true 1153 Acked 2152 Write 10 Done Done New true 1154 Acked 2153 Write 2 Done Done New true 1155 Acked 2154 Read 6 Done Done New true 1156 Acked 2155 Write 10 Done Done New true 1157 Acked 2156 Read 5 Done Done New true 1158 Acked 2157 Write 8 Done Done New true 1159 Acked 2158 Read 4 Done Done New true 1160 Acked 2159 Read 1 Done Done New true 1161 Acked 2160 Write 9 Done Done New true 1162 Acked 2161 Read 1 Done Done New true 1163 Acked 2162 Read 10 Done Done New true 1164 Acked 2163 Read 7 Done Done New true 1165 Acked 2164 Read 6 Done Done New true 1166 Acked 2165 Read 10 Done Done New true 1167 Acked 2166 Write 7 Done Done New true 1168 Acked 2167 Flush 0 Done Done New true 1169 Acked 2168 Flush 0 Done Done New true 1170 Acked 2169 Read 3 Done Done New true 1171 Acked 2170 Flush 0 Done Done New true 1172 Acked 2171 Read 4 Done Done New true 1173 Acked 2172 Write 4 Done Done New true 1174 Acked 2173 Write 10 Done Done New true 1175 Acked 2174 Write 7 Done Done New true 1176 Acked 2175 Flush 0 Done Done New true 1177 Acked 2176 Write 4 Done Done New true 1178 Acked 2177 Write 2 Done Done New true 1179 Acked 2178 Read 3 Done Done New true 1180 Acked 2179 Read 10 Done Done New true 1181 Acked 2180 Read 5 Done Done New true 1182 Acked 2181 Write 1 Done Done New true 1183 Acked 2182 Read 5 Done Done New true 1184 Acked 2183 Read 10 Done Done New true 1185 Acked 2184 Flush 0 Done Done New true 1186 Acked 2185 Read 9 Done Done New true 1187 Acked 2186 Read 5 Done Done New true 1188 Acked 2187 Write 7 Done Done New true 1189 Acked 2188 Read 1 Done Done New true 1190 Acked 2189 Write 2 Done Done New true 1191 Acked 2190 Write 1 Done Done New true 1192 Acked 2191 Flush 0 Done Done New true 1193 Acked 2192 Read 1 Done Done New true 1194 Acked 2193 Write 7 Done Done New true 1195 Acked 2194 Flush 0 Done Done New true 1196 Acked 2195 Flush 0 Done Done New true 1197 Acked 2196 Read 1 Done Done New true 1198 Acked 2197 Write 5 Done Done New true 1199 Acked 2198 Write 6 Done Done New true 1200 Acked 2199 Read 7 Done Done New true 1201 Acked 2200 Write 7 Done Done New true 1202 Acked 2201 Read 1 Done Done New true 1203 Acked 2202 Read 6 Done Done New true 1204 Acked 2203 Read 1 Done Done New true 1205 Acked 2204 Read 2 Done Done New true 1206 Acked 2205 Read 4 Done Done New true 1207 Acked 2206 Flush 0 Done Done New true STATES DS:0 DS:1 DS:2 TOTAL New 0 0 302 302 Sent 0 0 0 0 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2206 2206 1904 Downstairs last five completed: 1904 1903 1902 1901 1900 Upstairs last five completed: 1207 1206 1205 1204 1203 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:14.079224091Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:14.079290888Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:14.079318951Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:14.079354781Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:14.0793761Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:14.079399097Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:14.079652735Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:15.080515419Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:15.080586513Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:15.080615331Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:15.080641987Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:15.080662514Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:15.080694768Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:15.080960642Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1189) [8830][2] reports Running {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.081977764Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.08209742Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084121249Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084157559Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084181066Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 1904","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084468651Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084502955Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084526338Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084548862Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:16.084571097Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 1207 1206 1205 1204 1203 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(1) stop 1 Jan 02 22:31:17.934 INFO request completed, latency_us: 430, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4dbcdb0a-d3c9-48ac-a9ea-12915cf112ee, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 [8820] Got stop action so:false kr:false Jan 02 22:31:17.935 INFO request completed, latency_us: 274, response_code: 200, uri: /state/cid/1, method: GET, req_id: 59608ec4-e167-4531-a773-2b78cdd9a81d, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:17.936169896Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:17.936257332Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:17.936304687Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:17.936358822Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:17.936404136Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:17.936458696Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:18.937361276Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:18.937427934Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:18.93745607Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:18.93748283Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:18.937503222Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:18.937535387Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:18.937637037Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:19.938610775Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:19.938679601Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:19.938707021Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:19.938733191Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:19.938763862Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:19.938806736Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:19.938904658Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:20.939721734Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:20.93979154Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:20.939819958Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:20.939846517Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:20.939866719Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:20.939907302Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:20.939968399Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:31:21.937 INFO request completed, latency_us: 258, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6c832be8-b8b3-41a5-8ef8-22e4e54e0ca8, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 2763 len 8192 002/300 Write block 1286 len 24576 data: 2 2 2 2 2 2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:21.940376733Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:21.940469657Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:21.94051591Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:21.940541402Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:21.940567662Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:21.940596407Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 1 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:21.940637672Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:21.940898377Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Write block 3992 len 16384 data: 3 4 5 5 004/300 Read block 1256 len 4096 005/300 Read block 4519 len 36864 006/300 Write block 1788 len 36864 data: 2 2 2 2 2 2 3 3 2 007/300 Write block 4907 len 36864 data: 3 3 3 3 3 3 3 3 2 008/300 Write block 2443 len 20480 data: 3 3 4 4 4 009/300 Write block 409 len 16384 data: 3 3 3 3 010/300 Write block 2043 len 32768 data: 3 3 2 2 4 4 2 2 011/300 Write block 1762 len 12288 data: 2 2 2 012/300 Write block 323 len 12288 data: 2 2 2 013/300 Read block 1023 len 36864 014/300 Write block 2612 len 20480 data: 2 3 3 3 2 015/300 Write block 3032 len 32768 data: 3 3 3 3 2 2 2 3 016/300 Write block 4535 len 24576 data: 4 4 3 3 3 3 017/300 Read block 2495 len 36864 018/300 Read block 1713 len 40960 019/300 Read block 3970 len 36864 020/300 Read block 1564 len 12288 021/300 Write block 32 len 20480 data: 2 2 2 2 2 022/300 Read block 3031 len 4096 023/300 Write block 3881 len 28672 data: 2 3 3 3 3 3 3 024/300 Write block 4869 len 8192 data: 4 3 025/300 Read block 539 len 24576 026/300 Flush 027/300 Write block 3047 len 8192 data: 3 3 028/300 Read block 594 len 12288 029/300 Write block 339 len 20480 data: 2 2 3 3 2 030/300 Read block 3484 len 36864 031/300 Write block 4885 len 12288 data: 3 3 3 032/300 Write block 633 len 12288 data: 3 3 3 033/300 Read block 1668 len 12288 034/300 Read block 4858 len 4096 035/300 Write block 1925 len 8192 data: 2 2 036/300 Write block 1547 len 8192 data: 2 3 037/300 Read block 3233 len 28672 038/300 Write block 213 len 24576 data: 2 2 2 2 2 3 039/300 Write block 3527 len 4096 data: 2 040/300 Write block 3891 len 36864 data: 2 2 2 2 3 3 4 4 3 041/300 Flush 042/300 Read block 632 len 28672 043/300 Read block 2608 len 32768 044/300 Write block 436 len 32768 data: 3 4 4 4 4 3 3 3 045/300 Read block 1040 len 40960 046/300 Read block 1762 len 28672 047/300 Write block 3480 len 4096 data: 2 048/300 Read block 2064 len 28672 049/300 Read block 379 len 32768 050/300 Read block 3275 len 24576 051/300 Read block 722 len 20480 052/300 Write block 1121 len 36864 data: 2 2 2 2 2 2 2 2 2 053/300 Write block 3172 len 36864 data: 2 2 2 2 2 2 3 3 3 054/300 Read block 1481 len 40960 055/300 Read block 3463 len 32768 056/300 Write block 1098 len 40960 data: 2 3 3 3 3 3 3 4 4 4 057/300 Read block 584 len 28672 058/300 Write block 3010 len 36864 data: 2 2 2 2 2 2 2 2 3 059/300 Flush 060/300 Write block 367 len 4096 data: 4 061/300 Read block 1247 len 36864 062/300 Write block 1257 len 20480 data: 3 3 2 2 2 063/300 Read block 1003 len 12288 064/300 Read block 2662 len 20480 065/300 Flush 066/300 Read block 4220 len 4096 067/300 Write block 3821 len 4096 data: 2 068/300 Write block 3984 len 32768 data: 2 2 2 2 2 2 3 3 069/300 Read block 1410 len 24576 070/300 Read block 3427 len 32768 071/300 Write block 653 len 40960 data: 2 2 2 3 3 3 3 2 2 2 072/300 Write block 2627 len 8192 data: 2 2 073/300 Write block 4144 len 8192 data: 3 4 074/300 Write block 1310 len 36864 data: 2 2 2 2 2 2 2 2 2 075/300 Flush 076/300 Write block 3048 len 40960 data: 4 2 3 3 3 3 3 3 3 3 077/300 Read block 6 len 40960 078/300 Read block 54 len 28672 079/300 Write block 1021 len 12288 data: 2 2 2 080/300 Flush 081/300 Read block 3108 len 24576 082/300 Read block 4008 len 12288 083/300 Write block 2240 len 4096 data: 2 084/300 Write block 2416 len 24576 data: 4 4 4 4 4 3 085/300 Flush 086/300 Write block 4671 len 20480 data: 2 2 2 2 2 087/300 Read block 1866 len 28672 088/300 Write block 2821 len 40960 data: 3 3 3 3 3 3 3 4 2 2 089/300 Write block 948 len 16384 data: 3 4 5 5 090/300 Write block 4266 len 32768 data: 3 3 4 4 4 3 3 2 091/300 Write block 3137 len 8192 data: 4 4 092/300 Write block 1675 len 20480 data: 3 4 3 3 3 093/300 Write block 938 len 24576 data: 4 4 3 3 3 3 094/300 Read block 622 len 32768 095/300 Flush 096/300 Read block 2996 len 16384 097/300 Write block 1888 len 4096 data: 2 098/300 Write block 2140 len 40960 data: 3 3 3 3 3 3 3 3 3 2 099/300 Read block 1001 len 8192 100/300 Write block 1745 len 12288 data: 4 4 4 101/300 Read block 4062 len 24576 102/300 Read block 43 len 4096 103/300 Read block 2567 len 28672 104/300 Write block 1683 len 12288 data: 2 2 2 105/300 Read block 1884 len 8192 106/300 Write block 2568 len 16384 data: 3 2 2 2 107/300 Read block 548 len 36864 108/300 Write block 4734 len 20480 data: 4 4 4 4 4 109/300 Read block 1700 len 24576 110/300 Write block 490 len 24576 data: 2 2 2 2 2 2 111/300 Write block 1966 len 40960 data: 2 2 2 3 3 3 3 3 3 2 112/300 Write block 4094 len 36864 data: 2 2 3 3 3 3 3 3 3 113/300 Read block 221 len 16384 114/300 Read block 3961 len 36864 115/300 Write block 1224 len 8192 data: 3 3 116/300 Read block 727 len 8192 117/300 Write block 23 len 12288 data: 2 2 2 118/300 Read block 897 len 24576 119/300 Write block 332 len 12288 data: 2 2 2 120/300 Read block 3542 len 4096 121/300 Write block 3177 len 4096 data: 3 122/300 Write block 801 len 12288 data: 2 2 2 123/300 Read block 4481 len 36864 124/300 Write block 3588 len 32768 data: 2 2 2 2 2 2 2 2 125/300 Read block 1315 len 28672 126/300 Read block 836 len 4096 127/300 Read block 3629 len 16384 128/300 Write block 707 len 12288 data: 2 2 2 129/300 Write block 4625 len 24576 data: 2 2 2 2 2 2 130/300 Read block 1956 len 40960 131/300 Read block 747 len 20480 132/300 Read block 660 len 32768 133/300 Read block 3979 len 8192 134/300 Flush 135/300 Read block 3056 len 16384 136/300 Read block 4610 len 16384 137/300 Write block 546 len 24576 data: 2 2 2 2 2 2 138/300 Write block 353 len 12288 data: 2 2 2 139/300 Write block 4639 len 24576 data: 4 4 3 2 3 3 140/300 Write block 4923 len 24576 data: 2 2 3 3 4 4 141/300 Read block 653 len 36864 142/300 Write block 1737 len 32768 data: 2 2 2 2 2 2 4 4 143/300 Write block 2548 len 32768 data: 3 3 3 2 2 2 2 2 144/300 Write block 2471 len 40960 data: 2 2 2 2 2 2 2 2 3 3 145/300 Write block 1736 len 4096 data: 2 146/300 Write block 2160 len 24576 data: 3 2 2 2 2 2 147/300 Write block 539 len 4096 data: 3 148/300 Read block 1238 len 24576 149/300 Write block 3252 len 24576 data: 3 3 3 3 3 3 150/300 Flush 151/300 Read block 4223 len 12288 152/300 Write block 1096 len 36864 data: 3 2 3 4 4 4 4 4 4 153/300 Read block 4198 len 32768 154/300 Write block 2774 len 16384 data: 3 2 2 2 155/300 Flush 156/300 Read block 2790 len 16384 157/300 Write block 2275 len 40960 data: 3 3 3 2 2 2 2 2 2 2 158/300 Write block 2007 len 28672 data: 3 4 4 4 3 3 2 159/300 Flush 160/300 Read block 4387 len 40960 161/300 Write block 2825 len 24576 data: 4 4 4 5 3 3 162/300 Read block 688 len 28672 163/300 Write block 705 len 40960 data: 2 2 3 3 3 2 2 2 2 2 164/300 Read block 1165 len 20480 165/300 Write block 4664 len 16384 data: 2 2 2 2 166/300 Read block 1997 len 36864 167/300 Read block 3293 len 4096 168/300 Read block 1011 len 16384 169/300 Read block 3176 len 4096 170/300 Write block 3465 len 24576 data: 2 2 2 2 2 2 171/300 Read block 3803 len 16384 172/300 Read block 3943 len 40960 173/300 Write block 2005 len 32768 data: 3 3 4 5 5 5 4 4 174/300 Write block 2581 len 4096 data: 2 175/300 Read block 1239 len 24576 176/300 Write block 4045 len 12288 data: 4 4 4 177/300 Read block 4956 len 20480 178/300 Read block 4165 len 28672 179/300 Read block 3784 len 32768 180/300 Write block 2644 len 12288 data: 2 2 2 181/300 Read block 1689 len 36864 182/300 Write block 2746 len 40960 data: 2 2 2 2 2 2 3 3 3 3 183/300 Read block 4789 len 36864 184/300 Read block 267 len 12288 185/300 Read block 2990 len 4096 186/300 Read block 4274 len 4096 187/300 Read block 2039 len 16384 188/300 Read block 363 len 4096 189/300 Flush 190/300 Read block 4493 len 28672 191/300 Read block 2764 len 12288 192/300 Flush 193/300 Read block 1434 len 24576 194/300 Write block 866 len 28672 data: 2 2 2 2 2 2 2 195/300 Read block 63 len 24576 196/300 Read block 3983 len 36864 197/300 Read block 4648 len 12288 198/300 Write block 4172 len 32768 data: 2 2 2 2 2 2 2 2 199/300 Read block 1187 len 16384 200/300 Write block 3600 len 16384 data: 2 2 3 3 201/300 Flush 202/300 Read block 4070 len 24576 203/300 Read block 1242 len 8192 204/300 Write block 3881 len 32768 data: 3 4 4 4 4 4 4 3 205/300 Read block 2574 len 12288 206/300 Read block 1422 len 40960 207/300 Read block 959 len 32768 208/300 Read block 2788 len 36864 209/300 Write block 1755 len 40960 data: 3 3 3 3 3 2 2 3 3 3 210/300 Write block 4918 len 8192 data: 2 2 211/300 Read block 3673 len 32768 212/300 Write block 20 len 36864 data: 3 2 2 3 3 3 2 2 2 213/300 Write block 1999 len 24576 data: 2 2 3 3 3 3 214/300 Read block 708 len 24576 215/300 Write block 690 len 16384 data: 2 2 2 2 216/300 Read block 2999 len 4096 217/300 Flush 218/300 Write block 4333 len 20480 data: 2 2 2 2 2 219/300 Write block 3277 len 12288 data: 2 2 2 220/300 Read block 154 len 20480 221/300 Read block 1719 len 20480 222/300 Read block 97 len 40960 223/300 Read block 892 len 40960 224/300 Write block 3723 len 36864 data: 4 4 4 4 3 3 3 3 3 225/300 Read block 2337 len 28672 226/300 Write block 3359 len 28672 data: 4 4 4 4 4 3 3 227/300 Write block 694 len 12288 data: 2 2 3 228/300 Write block 1270 len 24576 data: 2 2 2 2 3 3 229/300 Read block 4064 len 32768 230/300 Read block 2092 len 16384 231/300 Write block 4375 len 28672 data: 2 2 2 2 3 3 3 232/300 Write block 1918 len 12288 data: 2 2 2 233/300 Flush 234/300 Read block 2843 len 4096 235/300 Read block 4646 len 36864 236/300 Flush 237/300 Write block 2363 len 16384 data: 4 4 4 3 238/300 Read block 3701 len 28672 239/300 Read block 2162 len 40960 240/300 Read block 3654 len 4096 241/300 Read block 4429 len 24576 242/300 Read block 1771 len 28672 243/300 Read block 4804 len 8192 244/300 Write block 3095 len 32768 data: 2 2 2 2 2 2 2 2 245/300 Read block 2215 len 28672 246/300 Read block 2385 len 4096 247/300 Read block 2033 len 12288 248/300 Read block 724 len 12288 249/300 Write block 1786 len 8192 data: 2 2 250/300 Read block 2588 len 36864 251/300 Read block 944 len 28672 252/300 Flush 253/300 Read block 325 len 40960 254/300 Write block 1977 len 32768 data: 3 3 3 2 2 2 2 2 255/300 Read block 1173 len 12288 256/300 Write block 308 len 36864 data: 3 3 4 4 4 3 2 2 2 257/300 Read block 1664 len 20480 258/300 Write block 3745 len 8192 data: 3 3 259/300 Write block 3536 len 40960 data: 2 2 2 2 2 2 2 2 2 2 260/300 Write block 3537 len 28672 data: 3 3 3 3 3 3 3 261/300 Read block 2471 len 20480 262/300 Write block 959 len 16384 data: 2 2 2 2 263/300 Read block 2020 len 24576 264/300 Write block 3031 len 8192 data: 3 4 265/300 Flush 266/300 Read block 1473 len 24576 267/300 Write block 3455 len 36864 data: 2 2 2 2 2 2 2 2 2 268/300 Write block 2544 len 36864 data: 2 2 3 3 4 4 4 3 3 269/300 Flush 270/300 Read block 1917 len 12288 271/300 Read block 1402 len 36864 272/300 Read block 4652 len 24576 273/300 Write block 2741 len 16384 data: 3 3 2 2 274/300 Read block 4126 len 24576 275/300 Read block 4876 len 32768 276/300 Read block 935 len 8192 277/300 Write block 2048 len 8192 data: 5 3 278/300 Read block 1921 len 36864 279/300 Write block 1943 len 24576 data: 2 2 2 2 2 2 280/300 Read block 2584 len 40960 281/300 Read block 634 len 4096 282/300 Read block 1883 len 24576 283/300 Flush 284/300 Write block 2948 len 36864 data: 4 4 4 4 3 3 3 2 2 285/300 Read block 4286 len 4096 286/300 Write block 377 len 16384 data: 4 4 4 4 287/300 Write block 1680 len 36864 data: 3 3 2 3 3 3 2 2 2 288/300 Read block 4565 len 8192 289/300 Write block 313 len 12288 data: 4 3 3 290/300 Write block 4372 len 40960 data: 2 2 2 3 3 3 3 4 4 4 291/300 Write block 4770 len 28672 data: 3 3 3 3 2 2 2 292/300 Read block 3279 len 16384 293/300 Flush 294/300 Read block 2586 len 12288 295/300 Flush 296/300 Write block 663 len 16384 data: 4 4 4 4 297/300 Write block 1912 len 36864 data: 3 2 2 2 2 2 3 3 3 298/300 Write block 1349 len 8192 data: 4 5 299/300 Read block 2586 len 20480 300/300 Write block 2670 len 40960 data: 2 2 2 2 2 2 2 2 2 2 Main task has work to do, go find it got dsc Start(1) start 1 Jan 02 22:31:22.918 INFO request completed, latency_us: 366, response_code: 204, uri: /start/cid/1, method: POST, req_id: 54c22900-e8a3-4f7c-8ea3-9cb867b9bc20, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1208 Acked 2207 Read 2 Done New Done true 1209 Acked 2208 Write 6 Done New Done false 1210 Acked 2209 Write 4 Done New Done false 1211 Acked 2210 Read 1 Done New Done false 1212 Acked 2211 Read 9 Done New Done false 1213 Acked 2212 Write 9 Done New Done false 1214 Acked 2213 Write 9 Done New Done false 1215 Acked 2214 Write 5 Done New Done false 1216 Acked 2215 Write 4 Done New Done false 1217 Acked 2216 Write 8 Done New Done false 1218 Acked 2217 Write 3 Done New Done false 1219 Acked 2218 Write 3 Done New Done false 1220 Acked 2219 Read 9 Done New Done false 1221 Acked 2220 Write 5 Done New Done false 1222 Acked 2221 Write 8 Done New Done false 1223 Acked 2222 Write 6 Done New Done false 1224 Acked 2223 Read 9 Done New Done false 1225 Acked 2224 Read 10 Done New Done false 1226 Acked 2225 Read 9 Done New Done false 1227 Acked 2226 Read 3 Done New Done false 1228 Acked 2227 Write 5 Done New Done false 1229 Acked 2228 Read 1 Done New Done false 1230 Acked 2229 Write 7 Done New Done false 1231 Acked 2230 Write 2 Done New Done false 1232 Acked 2231 Read 6 Done New Done false 1233 Acked 2232 Flush 0 Done New Done false 1234 Acked 2233 Write 2 Done New Done false 1235 Acked 2234 Read 3 Done New Done false 1236 Acked 2235 Write 5 Done New Done false 1237 Acked 2236 Read 9 Done New Done false 1238 Acked 2237 Write 3 Done New Done false 1239 Acked 2238 Write 3 Done New Done false 1240 Acked 2239 Read 3 Done New Done false 1241 Acked 2240 Read 1 Done New Done false 1242 Acked 2241 Write 2 Done New Done false 1243 Acked 2242 Write 2 Done New Done false 1244 Acked 2243 Read 7 Done New Done false 1245 Acked 2244 Write 6 Done New Done false 1246 Acked 2245 Write 1 Done New Done false 1247 Acked 2246 Write 9 Done New Done false 1248 Acked 2247 Flush 0 Done New Done false 1249 Acked 2248 Read 7 Done New Done false 1250 Acked 2249 Read 8 Done New Done false 1251 Acked 2250 Write 8 Done New Done false 1252 Acked 2251 Read 10 Done New Done false 1253 Acked 2252 Read 7 Done New Done false 1254 Acked 2253 Write 1 Done New Done false 1255 Acked 2254 Read 7 Done New Done false 1256 Acked 2255 Read 8 Done New Done false 1257 Acked 2256 Read 6 Done New Done false 1258 Acked 2257 Read 5 Done New Done false 1259 Acked 2258 Write 9 Done New Done false 1260 Acked 2259 Write 9 Done New Done false 1261 Acked 2260 Read 10 Done New Done false 1262 Acked 2261 Read 8 Done New Done false 1263 Acked 2262 Write 10 Done New Done false 1264 Acked 2263 Read 7 Done New Done false 1265 Acked 2264 Write 9 Done New Done false 1266 Acked 2265 Flush 0 Done New Done false 1267 Acked 2266 Write 1 Done New Done false 1268 Acked 2267 Read 9 Done New Done false 1269 Acked 2268 Write 5 Done New Done false 1270 Acked 2269 Read 3 Done New Done false 1271 Acked 2270 Read 5 Done New Done false 1272 Acked 2271 Flush 0 Done New Done false 1273 Acked 2272 Read 1 Done New Done false 1274 Acked 2273 Write 1 Done New Done false 1275 Acked 2274 Write 8 Done New Done false 1276 Acked 2275 Read 6 Done New Done false 1277 Acked 2276 Read 8 Done New Done false 1278 Acked 2277 Write 10 Done New Done false 1279 Acked 2278 Write 2 Done New Done false 1280 Acked 2279 Write 2 Done New Done false 1281 Acked 2280 Write 9 Done New Done false 1282 Acked 2281 Flush 0 Done New Done false 1283 Acked 2282 Write 10 Done New Done false 1284 Acked 2283 Read 10 Done New Done false 1285 Acked 2284 Read 7 Done New Done false 1286 Acked 2285 Write 3 Done New Done false 1287 Acked 2286 Flush 0 Done New Done false 1288 Acked 2287 Read 6 Done New Done false 1289 Acked 2288 Read 3 Done New Done false 1290 Acked 2289 Write 1 Done New Done false 1291 Acked 2290 Write 6 Done New Done false 1292 Acked 2291 Flush 0 Done New Done false 1293 Acked 2292 Write 5 Done New Done false 1294 Acked 2293 Read 7 Done New Done false 1295 Acked 2294 Write 10 Done New Done false 1296 Acked 2295 Write 4 Done New Done false 1297 Acked 2296 Write 8 Done New Done false 1298 Acked 2297 Write 2 Done New Done false 1299 Acked 2298 Write 5 Done New Done false 1300 Acked 2299 Write 6 Done New Done false 1301 Acked 2300 Read 8 Done New Done false 1302 Acked 2301 Flush 0 Done New Done false 1303 Acked 2302 Read 4 Done New Done false 1304 Acked 2303 Write 1 Done New Done false 1305 Acked 2304 Flush 0 Done New Done false 1306 Acked 2305 Write 10 Done New Done false 1307 Acked 2306 Read 2 Done New Done false 1308 Acked 2307 Write 3 Done New Done false 1309 Acked 2308 Read 6 Done New Done false 1310 Acked 2309 Read 1 Done New Done false 1311 Acked 2310 Read 7 Done New Done false 1312 Acked 2311 Write 3 Done New Done false 1313 Acked 2312 Read 2 Done New Done false 1314 Acked 2313 Write 4 Done New Done false 1315 Acked 2314 Read 9 Done New Done false 1316 Acked 2315 Write 5 Done New Done false 1317 Acked 2316 Read 6 Done New Done false 1318 Acked 2317 Write 6 Done New Done false 1319 Acked 2318 Write 10 Done New Done false 1320 Acked 2319 Write 9 Done New Done false 1321 Acked 2320 Read 4 Done New Done false 1322 Acked 2321 Read 9 Done New Done false 1323 Acked 2322 Write 2 Done New Done false 1324 Acked 2323 Read 2 Done New Done false 1325 Acked 2324 Write 3 Done New Done false 1326 Acked 2325 Read 6 Done New Done false 1327 Acked 2326 Write 3 Done New Done false 1328 Acked 2327 Read 1 Done New Done false 1329 Acked 2328 Write 1 Done New Done false 1330 Acked 2329 Write 3 Done New Done false 1331 Acked 2330 Read 9 Done New Done false 1332 Acked 2331 Write 8 Done New Done false 1333 Acked 2332 Read 7 Done New Done false 1334 Acked 2333 Read 1 Done New Done false 1335 Acked 2334 Read 4 Done New Done false 1336 Acked 2335 Write 3 Done New Done false 1337 Acked 2336 Write 6 Done New Done false 1338 Acked 2337 Read 10 Done New Done false 1339 Acked 2338 Read 5 Done New Done false 1340 Acked 2339 Read 8 Done New Done false 1341 Acked 2340 Read 2 Done New Done false 1342 Acked 2341 Flush 0 Done New Done false 1343 Acked 2342 Read 4 Done New Done false 1344 Acked 2343 Read 4 Done New Done false 1345 Acked 2344 Write 6 Done New Done false 1346 Acked 2345 Write 3 Done New Done false 1347 Acked 2346 Write 6 Done New Done false 1348 Acked 2347 Write 6 Done New Done false 1349 Acked 2348 Read 9 Done New Done false 1350 Acked 2349 Write 8 Done New Done false 1351 Acked 2350 Write 8 Done New Done false 1352 Acked 2351 Write 10 Done New Done false 1353 Acked 2352 Write 1 Done New Done false 1354 Acked 2353 Write 6 Done New Done false 1355 Acked 2354 Write 1 Done New Done false 1356 Acked 2355 Read 6 Done New Done false 1357 Acked 2356 Write 6 Done New Done false 1358 Acked 2357 Flush 0 Done New Done false 1359 Acked 2358 Read 3 Done New Done false 1360 Acked 2359 Write 9 Done New Done false 1361 Acked 2360 Read 8 Done New Done false 1362 Acked 2361 Write 4 Done New Done false 1363 Acked 2362 Flush 0 Done New Done false 1364 Acked 2363 Read 4 Done New Done false 1365 Acked 2364 Write 10 Done New Done false 1366 Acked 2365 Write 7 Done New Done false 1367 Acked 2366 Flush 0 Done New Done false 1368 Acked 2367 Read 10 Done New Done false 1369 Acked 2368 Write 6 Done New Done false 1370 Acked 2369 Read 7 Done New Done false 1371 Acked 2370 Write 10 Done New Done false 1372 Acked 2371 Read 5 Done New Done false 1373 Acked 2372 Write 4 Done New Done false 1374 Acked 2373 Read 9 Done New Done false 1375 Acked 2374 Read 1 Done New Done false 1376 Acked 2375 Read 4 Done New Done false 1377 Acked 2376 Read 1 Done New Done false 1378 Acked 2377 Write 6 Done New Done false 1379 Acked 2378 Read 4 Done New Done false 1380 Acked 2379 Read 10 Done New Done false 1381 Acked 2380 Write 8 Done New Done false 1382 Acked 2381 Write 1 Done New Done false 1383 Acked 2382 Read 6 Done New Done false 1384 Acked 2383 Write 3 Done New Done false 1385 Acked 2384 Read 5 Done New Done false 1386 Acked 2385 Read 7 Done New Done false 1387 Acked 2386 Read 8 Done New Done false 1388 Acked 2387 Write 3 Done New Done false 1389 Acked 2388 Read 9 Done New Done false 1390 Acked 2389 Write 10 Done New Done false 1391 Acked 2390 Read 9 Done New Done false 1392 Acked 2391 Read 3 Done New Done false 1393 Acked 2392 Read 1 Done New Done false 1394 Acked 2393 Read 1 Done New Done false 1395 Acked 2394 Read 4 Done New Done false 1396 Acked 2395 Read 1 Done New Done false 1397 Acked 2396 Flush 0 Done New Done false 1398 Acked 2397 Read 7 Done New Done false 1399 Acked 2398 Read 3 Done New Done false 1400 Acked 2399 Flush 0 Done New Done false 1401 Acked 2400 Read 6 Done New Done false 1402 Acked 2401 Write 7 Done New Done false 1403 Acked 2402 Read 6 Done New Done false 1404 Acked 2403 Read 9 Done New Done false 1405 Acked 2404 Read 3 Done New Done false 1406 Acked 2405 Write 8 Done New Done false 1407 Acked 2406 Read 4 Done New Done false 1408 Acked 2407 Write 4 Done New Done false 1409 Acked 2408 Flush 0 Done New Done false 1410 Acked 2409 Read 6 Done New Done false 1411 Acked 2410 Read 2 Done New Done false 1412 Acked 2411 Write 8 Done New Done false 1413 Acked 2412 Read 3 Done New Done false 1414 Acked 2413 Read 10 Done New Done false 1415 Acked 2414 Read 8 Done New Done false 1416 Acked 2415 Read 9 Done New Done false 1417 Acked 2416 Write 10 Done New Done false 1418 Acked 2417 Write 2 Done New Done false 1419 Acked 2418 Read 8 Done New Done false 1420 Acked 2419 Write 9 Done New Done false 1421 Acked 2420 Write 6 Done New Done false 1422 Acked 2421 Read 6 Done New Done false 1423 Acked 2422 Write 4 Done New Done false 1424 Acked 2423 Read 1 Done New Done false 1425 Acked 2424 Flush 0 Done New Done false 1426 Acked 2425 Write 5 Done New Done false 1427 Acked 2426 Write 3 Done New Done false 1428 Acked 2427 Read 5 Done New Done false 1429 Acked 2428 Read 5 Done New Done false 1430 Acked 2429 Read 10 Done New Done false 1431 Acked 2430 Read 10 Done New Done false 1432 Acked 2431 Write 9 Done New Done false 1433 Acked 2432 Read 7 Done New Done false 1434 Acked 2433 Write 7 Done New Done false 1435 Acked 2434 Write 3 Done New Done false 1436 Acked 2435 Write 6 Done New Done false 1437 Acked 2436 Read 8 Done New Done false 1438 Acked 2437 Read 4 Done New Done false 1439 Acked 2438 Write 7 Done New Done false 1440 Acked 2439 Write 3 Done New Done false 1441 Acked 2440 Flush 0 Done New Done false 1442 Acked 2441 Read 1 Done New Done false 1443 Acked 2442 Read 9 Done New Done false 1444 Acked 2443 Flush 0 Done New Done false 1445 Acked 2444 Write 4 Done New Done false 1446 Acked 2445 Read 7 Done New Done false 1447 Acked 2446 Read 10 Done New Done false 1448 Acked 2447 Read 1 Done New Done false 1449 Acked 2448 Read 6 Done New Done false 1450 Acked 2449 Read 7 Done New Done false 1451 Acked 2450 Read 2 Done New Done false 1452 Acked 2451 Write 8 Done New Done false 1453 Acked 2452 Read 7 Done New Done false 1454 Acked 2453 Read 1 Done New Done false 1455 Acked 2454 Read 3 Done New Done false 1456 Acked 2455 Read 3 Done New Done false 1457 Acked 2456 Write 2 Done New Done false 1458 Acked 2457 Read 9 Done New Done false 1459 Acked 2458 Read 7 Done New Done false 1460 Acked 2459 Flush 0 Done New Done false 1461 Acked 2460 Read 10 Done New Done false 1462 Acked 2461 Write 8 Done New Done false 1463 Acked 2462 Read 3 Done New Done false 1464 Acked 2463 Write 9 Done New Done false 1465 Acked 2464 Read 5 Done New Done false 1466 Acked 2465 Write 2 Done New Done false 1467 Acked 2466 Write 10 Done New Done false 1468 Acked 2467 Write 7 Done New Done false 1469 Acked 2468 Read 5 Done New Done false 1470 Acked 2469 Write 4 Done New Done false 1471 Acked 2470 Read 6 Done New Done false 1472 Acked 2471 Write 2 Done New Done false 1473 Acked 2472 Flush 0 Done New Done false 1474 Acked 2473 Flush 0 Done New Done false 1475 Acked 2474 Read 6 Done New Done false 1476 Acked 2475 Write 9 Done New Done false 1477 Acked 2476 Write 9 Done New Done false 1478 Acked 2477 Flush 0 Done New Done false 1479 Acked 2478 Read 3 Done New Done false 1480 Acked 2479 Read 9 Done New Done false 1481 Acked 2480 Read 6 Done New Done false 1482 Acked 2481 Write 4 Done New Done false 1483 Acked 2482 Read 6 Done New Done false 1484 Acked 2483 Read 8 Done New Done false 1485 Acked 2484 Read 2 Done New Done false 1486 Acked 2485 Write 2 Done New Done false 1487 Acked 2486 Read 9 Done New Done false 1488 Acked 2487 Write 6 Done New Done false 1489 Acked 2488 Read 10 Done New Done false 1490 Acked 2489 Read 1 Done New Done false 1491 Acked 2490 Read 6 Done New Done false 1492 Acked 2491 Flush 0 Done New Done false 1493 Acked 2492 Write 9 Done New Done false 1494 Acked 2493 Read 1 Done New Done false 1495 Acked 2494 Write 4 Done New Done false 1496 Acked 2495 Write 9 Done New Done false 1497 Acked 2496 Read 2 Done New Done false 1498 Acked 2497 Write 3 Done New Done false 1499 Acked 2498 Write 10 Done New Done false 1500 Acked 2499 Write 7 Done New Done false 1501 Acked 2500 Read 4 Done New Done false 1502 Acked 2501 Flush 0 Done New Done false 1503 Acked 2502 Read 3 Done New Done false 1504 Acked 2503 Flush 0 Done New Done false 1505 Acked 2504 Write 4 Done New Done false 1506 Acked 2505 Write 9 Done New Done false 1507 Acked 2506 Write 2 Done New Done false 1508 Acked 2507 Read 5 Done New Done false 1509 Acked 2508 Write 10 Done New Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 302 0 302 Sent 0 0 0 0 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2503 2206 2503 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 1509 1508 1507 1506 1505 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:22.942110245Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:22.942176272Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:22.94220474Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:22.94223139Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:22.942257974Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:22.942281797Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:22.94247452Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:23.943480059Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:23.94354849Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:23.943577125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:23.943603672Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:23.94362428Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:23.943674392Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:23.94391857Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:24.944803087Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:24.944870442Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:24.944898701Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:24.944935885Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:24.944957291Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:24.945057625Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:24.94529059Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:25.946077795Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:25.946145355Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:25.946173529Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:25.946200499Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:25.946221181Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:25.94634291Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:25.946574666Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1190) [8820][1] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1208 Acked 2207 Read 2 Done New Done true 1209 Acked 2208 Write 6 Done New Done true 1210 Acked 2209 Write 4 Done New Done true 1211 Acked 2210 Read 1 Done New Done true 1212 Acked 2211 Read 9 Done New Done true 1213 Acked 2212 Write 9 Done New Done true 1214 Acked 2213 Write 9 Done New Done true 1215 Acked 2214 Write 5 Done New Done true 1216 Acked 2215 Write 4 Done New Done true 1217 Acked 2216 Write 8 Done New Done true 1218 Acked 2217 Write 3 Done New Done true 1219 Acked 2218 Write 3 Done New Done true 1220 Acked 2219 Read 9 Done New Done true 1221 Acked 2220 Write 5 Done New Done true 1222 Acked 2221 Write 8 Done New Done true 1223 Acked 2222 Write 6 Done New Done true 1224 Acked 2223 Read 9 Done New Done true 1225 Acked 2224 Read 10 Done New Done true 1226 Acked 2225 Read 9 Done New Done true 1227 Acked 2226 Read 3 Done New Done true 1228 Acked 2227 Write 5 Done New Done true 1229 Acked 2228 Read 1 Done New Done true 1230 Acked 2229 Write 7 Done New Done true 1231 Acked 2230 Write 2 Done New Done true 1232 Acked 2231 Read 6 Done New Done true 1233 Acked 2232 Flush 0 Done New Done true 1234 Acked 2233 Write 2 Done New Done true 1235 Acked 2234 Read 3 Done New Done true 1236 Acked 2235 Write 5 Done New Done true 1237 Acked 2236 Read 9 Done New Done true 1238 Acked 2237 Write 3 Done New Done true 1239 Acked 2238 Write 3 Done New Done true 1240 Acked 2239 Read 3 Done New Done true 1241 Acked 2240 Read 1 Done New Done true 1242 Acked 2241 Write 2 Done New Done true 1243 Acked 2242 Write 2 Done New Done true 1244 Acked 2243 Read 7 Done New Done true 1245 Acked 2244 Write 6 Done New Done true 1246 Acked 2245 Write 1 Done New Done true 1247 Acked 2246 Write 9 Done New Done true 1248 Acked 2247 Flush 0 Done New Done true 1249 Acked 2248 Read 7 Done New Done true 1250 Acked 2249 Read 8 Done New Done true 1251 Acked 2250 Write 8 Done New Done true 1252 Acked 2251 Read 10 Done New Done true 1253 Acked 2252 Read 7 Done New Done true 1254 Acked 2253 Write 1 Done New Done true 1255 Acked 2254 Read 7 Done New Done true 1256 Acked 2255 Read 8 Done New Done true 1257 Acked 2256 Read 6 Done New Done true 1258 Acked 2257 Read 5 Done New Done true 1259 Acked 2258 Write 9 Done New Done true 1260 Acked 2259 Write 9 Done New Done true 1261 Acked 2260 Read 10 Done New Done true 1262 Acked 2261 Read 8 Done New Done true 1263 Acked 2262 Write 10 Done New Done true 1264 Acked 2263 Read 7 Done New Done true 1265 Acked 2264 Write 9 Done New Done true 1266 Acked 2265 Flush 0 Done New Done true 1267 Acked 2266 Write 1 Done New Done true 1268 Acked 2267 Read 9 Done New Done true 1269 Acked 2268 Write 5 Done New Done true 1270 Acked 2269 Read 3 Done New Done true 1271 Acked 2270 Read 5 Done New Done true 1272 Acked 2271 Flush 0 Done New Done true 1273 Acked 2272 Read 1 Done New Done true 1274 Acked 2273 Write 1 Done New Done true 1275 Acked 2274 Write 8 Done New Done true 1276 Acked 2275 Read 6 Done New Done true 1277 Acked 2276 Read 8 Done New Done true 1278 Acked 2277 Write 10 Done New Done true 1279 Acked 2278 Write 2 Done New Done true 1280 Acked 2279 Write 2 Done New Done true 1281 Acked 2280 Write 9 Done New Done true 1282 Acked 2281 Flush 0 Done New Done true 1283 Acked 2282 Write 10 Done New Done true 1284 Acked 2283 Read 10 Done New Done true 1285 Acked 2284 Read 7 Done New Done true 1286 Acked 2285 Write 3 Done New Done true 1287 Acked 2286 Flush 0 Done New Done true 1288 Acked 2287 Read 6 Done New Done true 1289 Acked 2288 Read 3 Done New Done true 1290 Acked 2289 Write 1 Done New Done true 1291 Acked 2290 Write 6 Done New Done true 1292 Acked 2291 Flush 0 Done New Done true 1293 Acked 2292 Write 5 Done New Done true 1294 Acked 2293 Read 7 Done New Done true 1295 Acked 2294 Write 10 Done New Done true 1296 Acked 2295 Write 4 Done New Done true 1297 Acked 2296 Write 8 Done New Done true 1298 Acked 2297 Write 2 Done New Done true 1299 Acked 2298 Write 5 Done New Done true 1300 Acked 2299 Write 6 Done New Done true 1301 Acked 2300 Read 8 Done New Done true 1302 Acked 2301 Flush 0 Done New Done true 1303 Acked 2302 Read 4 Done New Done true 1304 Acked 2303 Write 1 Done New Done true 1305 Acked 2304 Flush 0 Done New Done true 1306 Acked 2305 Write 10 Done New Done true 1307 Acked 2306 Read 2 Done New Done true 1308 Acked 2307 Write 3 Done New Done true 1309 Acked 2308 Read 6 Done New Done true 1310 Acked 2309 Read 1 Done New Done true 1311 Acked 2310 Read 7 Done New Done true 1312 Acked 2311 Write 3 Done New Done true 1313 Acked 2312 Read 2 Done New Done true 1314 Acked 2313 Write 4 Done New Done true 1315 Acked 2314 Read 9 Done New Done true 1316 Acked 2315 Write 5 Done New Done true 1317 Acked 2316 Read 6 Done New Done true 1318 Acked 2317 Write 6 Done New Done true 1319 Acked 2318 Write 10 Done New Done true 1320 Acked 2319 Write 9 Done New Done true 1321 Acked 2320 Read 4 Done New Done true 1322 Acked 2321 Read 9 Done New Done true 1323 Acked 2322 Write 2 Done New Done true 1324 Acked 2323 Read 2 Done New Done true 1325 Acked 2324 Write 3 Done New Done true 1326 Acked 2325 Read 6 Done New Done true 1327 Acked 2326 Write 3 Done New Done true 1328 Acked 2327 Read 1 Done New Done true 1329 Acked 2328 Write 1 Done New Done true 1330 Acked 2329 Write 3 Done New Done true 1331 Acked 2330 Read 9 Done New Done true 1332 Acked 2331 Write 8 Done New Done true 1333 Acked 2332 Read 7 Done New Done true 1334 Acked 2333 Read 1 Done New Done true 1335 Acked 2334 Read 4 Done New Done true 1336 Acked 2335 Write 3 Done New Done true 1337 Acked 2336 Write 6 Done New Done true 1338 Acked 2337 Read 10 Done New Done true 1339 Acked 2338 Read 5 Done New Done true 1340 Acked 2339 Read 8 Done New Done true 1341 Acked 2340 Read 2 Done New Done true 1342 Acked 2341 Flush 0 Done New Done true 1343 Acked 2342 Read 4 Done New Done true 1344 Acked 2343 Read 4 Done New Done true 1345 Acked 2344 Write 6 Done New Done true 1346 Acked 2345 Write 3 Done New Done true 1347 Acked 2346 Write 6 Done New Done true 1348 Acked 2347 Write 6 Done New Done true 1349 Acked 2348 Read 9 Done New Done true 1350 Acked 2349 Write 8 Done New Done true 1351 Acked 2350 Write 8 Done New Done true 1352 Acked 2351 Write 10 Done New Done true 1353 Acked 2352 Write 1 Done New Done true 1354 Acked 2353 Write 6 Done New Done true 1355 Acked 2354 Write 1 Done New Done true 1356 Acked 2355 Read 6 Done New Done true 1357 Acked 2356 Write 6 Done New Done true 1358 Acked 2357 Flush 0 Done New Done true 1359 Acked 2358 Read 3 Done New Done true 1360 Acked 2359 Write 9 Done New Done true 1361 Acked 2360 Read 8 Done New Done true 1362 Acked 2361 Write 4 Done New Done true 1363 Acked 2362 Flush 0 Done New Done true 1364 Acked 2363 Read 4 Done New Done true 1365 Acked 2364 Write 10 Done New Done true 1366 Acked 2365 Write 7 Done New Done true 1367 Acked 2366 Flush 0 Done New Done true 1368 Acked 2367 Read 10 Done New Done true 1369 Acked 2368 Write 6 Done New Done true 1370 Acked 2369 Read 7 Done New Done true 1371 Acked 2370 Write 10 Done New Done true 1372 Acked 2371 Read 5 Done New Done true 1373 Acked 2372 Write 4 Done New Done true 1374 Acked 2373 Read 9 Done New Done true 1375 Acked 2374 Read 1 Done New Done true 1376 Acked 2375 Read 4 Done New Done true 1377 Acked 2376 Read 1 Done New Done true 1378 Acked 2377 Write 6 Done New Done true 1379 Acked 2378 Read 4 Done New Done true 1380 Acked 2379 Read 10 Done New Done true 1381 Acked 2380 Write 8 Done New Done true 1382 Acked 2381 Write 1 Done New Done true 1383 Acked 2382 Read 6 Done New Done true 1384 Acked 2383 Write 3 Done New Done true 1385 Acked 2384 Read 5 Done New Done true 1386 Acked 2385 Read 7 Done New Done true 1387 Acked 2386 Read 8 Done New Done true 1388 Acked 2387 Write 3 Done New Done true 1389 Acked 2388 Read 9 Done New Done true 1390 Acked 2389 Write 10 Done New Done true 1391 Acked 2390 Read 9 Done New Done true 1392 Acked 2391 Read 3 Done New Done true 1393 Acked 2392 Read 1 Done New Done true 1394 Acked 2393 Read 1 Done New Done true 1395 Acked 2394 Read 4 Done New Done true 1396 Acked 2395 Read 1 Done New Done true 1397 Acked 2396 Flush 0 Done New Done true 1398 Acked 2397 Read 7 Done New Done true 1399 Acked 2398 Read 3 Done New Done true 1400 Acked 2399 Flush 0 Done New Done true 1401 Acked 2400 Read 6 Done New Done true 1402 Acked 2401 Write 7 Done New Done true 1403 Acked 2402 Read 6 Done New Done true 1404 Acked 2403 Read 9 Done New Done true 1405 Acked 2404 Read 3 Done New Done true 1406 Acked 2405 Write 8 Done New Done true 1407 Acked 2406 Read 4 Done New Done true 1408 Acked 2407 Write 4 Done New Done true 1409 Acked 2408 Flush 0 Done New Done true 1410 Acked 2409 Read 6 Done New Done true 1411 Acked 2410 Read 2 Done New Done true 1412 Acked 2411 Write 8 Done New Done true 1413 Acked 2412 Read 3 Done New Done true 1414 Acked 2413 Read 10 Done New Done true 1415 Acked 2414 Read 8 Done New Done true 1416 Acked 2415 Read 9 Done New Done true 1417 Acked 2416 Write 10 Done New Done true 1418 Acked 2417 Write 2 Done New Done true 1419 Acked 2418 Read 8 Done New Done true 1420 Acked 2419 Write 9 Done New Done true 1421 Acked 2420 Write 6 Done New Done true 1422 Acked 2421 Read 6 Done New Done true 1423 Acked 2422 Write 4 Done New Done true 1424 Acked 2423 Read 1 Done New Done true 1425 Acked 2424 Flush 0 Done New Done true 1426 Acked 2425 Write 5 Done New Done true 1427 Acked 2426 Write 3 Done New Done true 1428 Acked 2427 Read 5 Done New Done true 1429 Acked 2428 Read 5 Done New Done true 1430 Acked 2429 Read 10 Done New Done true 1431 Acked 2430 Read 10 Done New Done true 1432 Acked 2431 Write 9 Done New Done true 1433 Acked 2432 Read 7 Done New Done true 1434 Acked 2433 Write 7 Done New Done true 1435 Acked 2434 Write 3 Done New Done true 1436 Acked 2435 Write 6 Done New Done true 1437 Acked 2436 Read 8 Done New Done true 1438 Acked 2437 Read 4 Done New Done true 1439 Acked 2438 Write 7 Done New Done true 1440 Acked 2439 Write 3 Done New Done true 1441 Acked 2440 Flush 0 Done New Done true 1442 Acked 2441 Read 1 Done New Done true 1443 Acked 2442 Read 9 Done New Done true 1444 Acked 2443 Flush 0 Done New Done true 1445 Acked 2444 Write 4 Done New Done true 1446 Acked 2445 Read 7 Done New Done true 1447 Acked 2446 Read 10 Done New Done true 1448 Acked 2447 Read 1 Done New Done true 1449 Acked 2448 Read 6 Done New Done true 1450 Acked 2449 Read 7 Done New Done true 1451 Acked 2450 Read 2 Done New Done true 1452 Acked 2451 Write 8 Done New Done true 1453 Acked 2452 Read 7 Done New Done true 1454 Acked 2453 Read 1 Done New Done true 1455 Acked 2454 Read 3 Done New Done true 1456 Acked 2455 Read 3 Done New Done true 1457 Acked 2456 Write 2 Done New Done true 1458 Acked 2457 Read 9 Done New Done true 1459 Acked 2458 Read 7 Done New Done true 1460 Acked 2459 Flush 0 Done New Done true 1461 Acked 2460 Read 10 Done New Done true 1462 Acked 2461 Write 8 Done New Done true 1463 Acked 2462 Read 3 Done New Done true 1464 Acked 2463 Write 9 Done New Done true 1465 Acked 2464 Read 5 Done New Done true 1466 Acked 2465 Write 2 Done New Done true 1467 Acked 2466 Write 10 Done New Done true 1468 Acked 2467 Write 7 Done New Done true 1469 Acked 2468 Read 5 Done New Done true 1470 Acked 2469 Write 4 Done New Done true 1471 Acked 2470 Read 6 Done New Done true 1472 Acked 2471 Write 2 Done New Done true 1473 Acked 2472 Flush 0 Done New Done true 1474 Acked 2473 Flush 0 Done New Done true 1475 Acked 2474 Read 6 Done New Done true 1476 Acked 2475 Write 9 Done New Done true 1477 Acked 2476 Write 9 Done New Done true 1478 Acked 2477 Flush 0 Done New Done true 1479 Acked 2478 Read 3 Done New Done true 1480 Acked 2479 Read 9 Done New Done true 1481 Acked 2480 Read 6 Done New Done true 1482 Acked 2481 Write 4 Done New Done true 1483 Acked 2482 Read 6 Done New Done true 1484 Acked 2483 Read 8 Done New Done true 1485 Acked 2484 Read 2 Done New Done true 1486 Acked 2485 Write 2 Done New Done true 1487 Acked 2486 Read 9 Done New Done true 1488 Acked 2487 Write 6 Done New Done true 1489 Acked 2488 Read 10 Done New Done true 1490 Acked 2489 Read 1 Done New Done true 1491 Acked 2490 Read 6 Done New Done true 1492 Acked 2491 Flush 0 Done New Done true 1493 Acked 2492 Write 9 Done New Done true 1494 Acked 2493 Read 1 Done New Done true 1495 Acked 2494 Write 4 Done New Done true 1496 Acked 2495 Write 9 Done New Done true 1497 Acked 2496 Read 2 Done New Done true 1498 Acked 2497 Write 3 Done New Done true 1499 Acked 2498 Write 10 Done New Done true 1500 Acked 2499 Write 7 Done New Done true 1501 Acked 2500 Read 4 Done New Done true 1502 Acked 2501 Flush 0 Done New Done true 1503 Acked 2502 Read 3 Done New Done true 1504 Acked 2503 Flush 0 Done New Done true 1505 Acked 2504 Write 4 Done New Done true 1506 Acked 2505 Write 9 Done New Done true 1507 Acked 2506 Write 2 Done New Done true 1508 Acked 2507 Read 5 Done New Done true 1509 Acked 2508 Write 10 Done New Done true 1510 Acked 2509 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 303 0 303 Sent 0 0 0 0 Done 303 0 303 606 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2509 2206 2509 Downstairs last five completed: 2206 2205 2204 2203 2202 Upstairs last five completed: 1510 1509 1508 1507 1506 CLIENT: Up:0 ds:303 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.947521136Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.947718993Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.949750649Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.949788196Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.949812061Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 2206","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.950103628Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.950142906Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.950166773Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.950188987Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:26.950211151Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 2509 2508 2507 2506 2505 Upstairs last five completed: 1510 1509 1508 1507 1506 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:31:30.925 INFO request completed, latency_us: 425, response_code: 204, uri: /stop/cid/0, method: POST, req_id[8810] Got stop action so:false kr:false : 6e876828-2f4f-4fc3-9efd-e797d8b77885, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:31:30.926 INFO request completed, latency_us: 301, response_code: 200, uri: /state/cid/0, method: GET, req_id: 473c51e4-ea65-44c1-bfbb-08b13675b93b, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:30.927254515Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:30.927341399Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:30.92737135Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:30.927393206Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:30.927415986Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:30.927461813Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:31.946429699Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:31.946495289Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:31.946529398Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:31.946553214Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:31.946577799Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:31.946597034Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:31.946618022Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:31.946656473Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:33.010158305Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:33.010223371Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:33.010249244Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:33.010274034Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:33.010293168Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:33.010337297Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:33.010432976Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:34.029360807Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:34.029427538Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:34.029455303Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:34.029481774Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:34.029502282Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:34.029547238Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:34.029665516Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:31:35.027 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9a553c79-af30-4ed9-9275-24019d053838, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 4372 len 8192 002/300 Read block 781 len 24576 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:35.030224768Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:35.030329071Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:35.03039228Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:35.030445135Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:35.030486932Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 2 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:35.030541756Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:35.030586401Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 1598 len 16384 004/300 Write block 267 len 36864 data: 3 4 4 4 4 4 4 3 2 005/300 Read block 1187 len 4096 006/300 Read block 1569 len 40960 007/300 Flush 008/300 Write block 3758 len 36864 data: 3 2 2 2 2 3 3 3 3 009/300 Write block 1831 len 8192 data: 2 2 010/300 Read block 2663 len 12288 011/300 Read block 2314 len 12288 012/300 Write block 1293 len 12288 data: 2 2 2 013/300 Flush 014/300 Flush 015/300 Write block 1985 len 12288 data: 2 3 3 016/300 Read block 2079 len 40960 017/300 Read block 3585 len 24576 018/300 Write block 594 len 20480 data: 2 2 2 3 3 019/300 Read block 3091 len 16384 020/300 Write block 4629 len 36864 data: 3 3 3 3 4 4 4 4 5 021/300 Write block 2985 len 36864 data: 2 2 2 2 3 3 3 2 2 022/300 Write block 4343 len 16384 data: 4 4 4 4 023/300 Read block 4133 len 36864 024/300 Flush 025/300 Write block 4983 len 16384 data: 4 4 3 3 026/300 Read block 1440 len 12288 027/300 Write block 2898 len 36864 data: 3 3 3 3 3 3 2 2 2 028/300 Read block 2230 len 12288 029/300 Read block 2718 len 12288 030/300 Read block 3562 len 32768 031/300 Flush 032/300 Flush 033/300 Write block 3590 len 36864 data: 3 3 3 3 3 3 2 2 2 034/300 Write block 3064 len 16384 data: 2 2 2 2 035/300 Flush 036/300 Write block 2684 len 20480 data: 2 2 2 3 3 037/300 Read block 3362 len 40960 038/300 Read block 4124 len 40960 039/300 Write block 2164 len 24576 data: 3 3 2 2 2 2 040/300 Write block 2141 len 4096 data: 4 041/300 Read block 2161 len 24576 042/300 Write block 2329 len 4096 data: 3 043/300 Read block 2884 len 24576 044/300 Write block 2835 len 40960 data: 2 2 2 2 2 2 2 2 2 2 045/300 Read block 720 len 12288 046/300 Read block 1013 len 24576 047/300 Read block 4103 len 8192 048/300 Write block 2843 len 24576 data: 3 3 2 2 2 2 049/300 Read block 3663 len 36864 050/300 Read block 2792 len 36864 051/300 Write block 1866 len 24576 data: 2 2 2 2 3 3 052/300 Read block 207 len 20480 053/300 Read block 2225 len 8192 054/300 Write block 3923 len 40960 data: 2 2 2 2 2 2 2 2 2 2 055/300 Write block 616 len 28672 data: 2 2 2 2 2 2 2 056/300 Read block 2926 len 8192 057/300 Write block 4957 len 28672 data: 2 2 2 2 2 2 2 058/300 Read block 3396 len 4096 059/300 Read block 2735 len 16384 060/300 Write block 3798 len 36864 data: 2 2 2 2 2 2 2 2 2 061/300 Write block 2337 len 16384 data: 3 2 2 2 062/300 Read block 4783 len 32768 063/300 Flush 064/300 Read block 4379 len 40960 065/300 Read block 2673 len 24576 066/300 Write block 26 len 40960 data: 3 3 3 2 2 2 3 3 3 3 067/300 Write block 3609 len 28672 data: 3 3 3 2 2 2 2 068/300 Write block 2461 len 36864 data: 3 3 3 3 3 3 2 2 2 069/300 Flush 070/300 Read block 999 len 16384 071/300 Flush 072/300 Write block 4787 len 20480 data: 2 2 2 2 2 073/300 Flush 074/300 Read block 4302 len 28672 075/300 Read block 3448 len 4096 076/300 Flush 077/300 Write block 3591 len 16384 data: 4 4 4 4 078/300 Write block 3766 len 4096 data: 4 079/300 Read block 3991 len 12288 080/300 Read block 232 len 4096 081/300 Write block 4118 len 36864 data: 2 2 2 2 2 2 2 2 2 082/300 Write block 254 len 36864 data: 2 2 2 2 2 2 2 2 2 083/300 Write block 648 len 36864 data: 3 2 2 2 2 3 3 3 4 084/300 Write block 2964 len 24576 data: 3 3 2 2 2 2 085/300 Write block 2008 len 40960 data: 6 6 6 5 5 3 2 2 2 2 086/300 Write block 981 len 28672 data: 2 2 2 2 3 2 2 087/300 Write block 2799 len 24576 data: 3 3 3 3 3 3 088/300 Write block 2231 len 4096 data: 2 089/300 Write block 1647 len 4096 data: 2 090/300 Write block 1823 len 20480 data: 2 2 2 2 2 091/300 Write block 4871 len 28672 data: 3 3 3 3 3 3 2 092/300 Write block 2582 len 36864 data: 2 2 2 2 2 2 2 2 2 093/300 Flush 094/300 Read block 477 len 36864 095/300 Read block 3471 len 8192 096/300 Read block 959 len 12288 097/300 Write block 670 len 8192 data: 3 3 098/300 Read block 2476 len 12288 099/300 Read block 867 len 32768 100/300 Read block 2383 len 8192 101/300 Read block 3322 len 28672 102/300 Read block 1872 len 28672 103/300 Read block 651 len 12288 104/300 Read block 2865 len 24576 105/300 Flush 106/300 Read block 3562 len 24576 107/300 Write block 3922 len 24576 data: 2 3 3 3 3 3 108/300 Write block 1084 len 20480 data: 2 2 2 2 2 109/300 Read block 4754 len 40960 110/300 Flush 111/300 Read block 4773 len 20480 112/300 Flush 113/300 Write block 4426 len 36864 data: 3 3 3 3 2 2 2 2 2 114/300 Read block 1136 len 28672 115/300 Write block 4432 len 12288 data: 3 3 3 116/300 Read block 1946 len 40960 117/300 Read block 2105 len 16384 118/300 Read block 4791 len 36864 119/300 Write block 1419 len 36864 data: 2 2 3 3 3 3 3 2 2 120/300 Read block 3607 len 40960 121/300 Read block 2857 len 24576 122/300 Write block 1323 len 36864 data: 2 2 3 3 3 2 2 2 2 123/300 Write block 1136 len 8192 data: 4 4 124/300 Flush 125/300 Read block 3619 len 4096 126/300 Read block 4682 len 12288 127/300 Write block 3930 len 16384 data: 3 3 3 2 128/300 Write block 2365 len 8192 data: 5 4 129/300 Read block 1432 len 20480 130/300 Flush 131/300 Read block 354 len 4096 132/300 Read block 242 len 4096 133/300 Read block 3279 len 4096 134/300 Write block 1355 len 12288 data: 3 2 2 135/300 Write block 1740 len 4096 data: 3 136/300 Read block 879 len 20480 137/300 Flush 138/300 Read block 2080 len 32768 139/300 Write block 4174 len 40960 data: 3 3 3 3 3 3 2 2 2 2 140/300 Read block 240 len 40960 141/300 Read block 3456 len 40960 142/300 Read block 1048 len 20480 143/300 Read block 1831 len 12288 144/300 Read block 1368 len 28672 145/300 Write block 1457 len 12288 data: 2 2 2 146/300 Read block 772 len 40960 147/300 Read block 1494 len 8192 148/300 Flush 149/300 Write block 4747 len 20480 data: 2 2 2 2 2 150/300 Read block 277 len 12288 151/300 Read block 1370 len 4096 152/300 Read block 1495 len 4096 153/300 Flush 154/300 Read block 3633 len 20480 155/300 Write block 2443 len 8192 data: 4 4 156/300 Flush 157/300 Write block 4732 len 40960 data: 4 5 5 5 5 5 5 3 3 3 158/300 Read block 2631 len 16384 159/300 Read block 2553 len 36864 160/300 Flush 161/300 Write block 4970 len 12288 data: 2 2 2 162/300 Read block 1633 len 20480 163/300 Write block 2268 len 8192 data: 3 4 164/300 Write block 4037 len 32768 data: 2 2 2 2 3 4 4 3 165/300 Flush 166/300 Read block 4765 len 4096 167/300 Read block 3474 len 8192 168/300 Write block 3914 len 8192 data: 4 4 169/300 Read block 2994 len 24576 170/300 Read block 4550 len 20480 171/300 Flush 172/300 Write block 796 len 12288 data: 3 3 2 173/300 Write block 3405 len 24576 data: 2 2 2 2 2 2 174/300 Write block 1723 len 40960 data: 2 2 2 2 2 2 2 2 2 2 175/300 Read block 2923 len 16384 176/300 Flush 177/300 Write block 2579 len 24576 data: 3 2 3 3 3 3 178/300 Read block 2392 len 20480 179/300 Write block 4137 len 20480 data: 2 2 2 2 2 180/300 Read block 2042 len 4096 181/300 Read block 2441 len 28672 182/300 Write block 2916 len 24576 data: 3 4 3 3 3 3 183/300 Write block 2853 len 16384 data: 2 2 2 2 184/300 Flush 185/300 Write block 3623 len 28672 data: 5 5 4 3 3 3 3 186/300 Read block 3903 len 8192 187/300 Write block 2728 len 8192 data: 2 2 188/300 Flush 189/300 Read block 2200 len 16384 190/300 Write block 1709 len 20480 data: 2 3 3 2 2 191/300 Read block 3741 len 24576 192/300 Write block 3763 len 36864 data: 4 4 4 5 3 3 3 3 3 193/300 Read block 1400 len 36864 194/300 Write block 1213 len 24576 data: 2 2 2 2 2 2 195/300 Read block 668 len 36864 196/300 Read block 1902 len 24576 197/300 Write block 1508 len 12288 data: 4 3 3 198/300 Write block 256 len 24576 data: 3 3 3 3 3 3 199/300 Read block 920 len 16384 200/300 Read block 135 len 32768 201/300 Read block 1943 len 12288 202/300 Read block 3639 len 28672 203/300 Flush 204/300 Read block 954 len 16384 205/300 Flush 206/300 Write block 3788 len 8192 data: 4 4 207/300 Write block 1149 len 4096 data: 2 208/300 Read block 1535 len 36864 209/300 Write block 3765 len 16384 data: 5 6 4 4 210/300 Write block 2455 len 36864 data: 2 2 2 3 3 3 4 4 4 211/300 Write block 1125 len 12288 data: 3 3 3 212/300 Write block 795 len 4096 data: 3 213/300 Read block 2818 len 20480 214/300 Write block 445 len 4096 data: 3 215/300 Flush 216/300 Read block 3574 len 28672 217/300 Read block 1352 len 8192 218/300 Write block 2388 len 40960 data: 2 2 2 2 2 2 2 2 2 2 219/300 Flush 220/300 Read block 1407 len 28672 221/300 Read block 4633 len 16384 222/300 Write block 4881 len 28672 data: 3 3 3 3 4 4 4 223/300 Flush 224/300 Write block 3374 len 24576 data: 2 2 2 2 2 2 225/300 Read block 80 len 24576 226/300 Write block 2572 len 36864 data: 2 3 3 3 3 3 3 4 3 227/300 Write block 4359 len 20480 data: 2 2 2 2 2 228/300 Flush 229/300 Read block 3318 len 28672 230/300 Flush 231/300 Read block 4197 len 12288 232/300 Read block 1453 len 24576 233/300 Write block 4571 len 8192 data: 2 2 234/300 Read block 3293 len 24576 235/300 Read block 3443 len 8192 236/300 Write block 2418 len 24576 data: 5 5 5 4 2 2 237/300 Read block 4762 len 4096 238/300 Write block 754 len 24576 data: 2 2 2 2 2 2 239/300 Read block 2607 len 8192 240/300 Read block 1043 len 4096 241/300 Write block 962 len 36864 data: 3 3 3 3 4 3 3 3 3 242/300 Write block 4844 len 24576 data: 4 3 3 2 2 2 243/300 Read block 1569 len 40960 244/300 Write block 828 len 8192 data: 2 2 245/300 Flush 246/300 Read block 2328 len 16384 247/300 Write block 675 len 8192 data: 2 2 248/300 Read block 1354 len 36864 249/300 Flush 250/300 Write block 4215 len 40960 data: 2 2 2 2 2 2 2 2 2 2 251/300 Write block 3344 len 32768 data: 3 3 3 3 3 3 3 2 252/300 Flush 253/300 Flush 254/300 Read block 125 len 12288 255/300 Write block 710 len 32768 data: 3 3 3 3 3 2 2 2 256/300 Read block 4266 len 40960 257/300 Read block 3629 len 36864 258/300 Read block 3144 len 20480 259/300 Write block 638 len 36864 data: 2 2 2 2 2 4 4 4 4 260/300 Read block 1604 len 36864 261/300 Read block 98 len 40960 262/300 Read block 579 len 12288 263/300 Write block 683 len 36864 data: 2 2 2 2 3 3 3 3 3 264/300 Flush 265/300 Read block 2375 len 12288 266/300 Read block 3794 len 20480 267/300 Read block 3372 len 36864 268/300 Read block 1407 len 4096 269/300 Read block 4670 len 36864 270/300 Read block 117 len 40960 271/300 Write block 3332 len 40960 data: 4 3 2 2 2 2 2 3 4 4 272/300 Read block 1157 len 8192 273/300 Flush 274/300 Read block 3238 len 12288 275/300 Write block 4173 len 4096 data: 3 276/300 Read block 1925 len 16384 277/300 Write block 822 len 24576 data: 2 2 2 2 2 2 278/300 Read block 4692 len 20480 279/300 Read block 374 len 8192 280/300 Read block 1667 len 8192 281/300 Read block 4542 len 16384 282/300 Read block 2282 len 8192 283/300 Write block 360 len 20480 data: 3 3 3 3 3 284/300 Read block 255 len 16384 285/300 Write block 1736 len 20480 data: 3 3 3 3 4 286/300 Read block 3677 len 36864 287/300 Write block 2768 len 12288 data: 5 6 7 288/300 Write block 1775 len 4096 data: 4 289/300 Read block 2986 len 24576 290/300 Write block 2208 len 16384 data: 2 3 3 3 291/300 Flush 292/300 Write block 1521 len 28672 data: 3 3 2 2 2 3 3 293/300 Read block 3520 len 40960 294/300 Read block 1262 len 12288 295/300 Read block 2914 len 16384 296/300 Read block 2334 len 20480 297/300 Read block 4410 len 16384 298/300 Read block 1002 len 24576 299/300 Flush 300/300 Read block 4675 len 28672 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:31:35.940 INFO request completed, latency_us: 283, response_code: 204, uri: /start/cid/0, method: POST, req_id: c75cf148-97f2-43cd-9370-4dd7076e8be6, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1511 Acked 2510 Read 2 New Done Done true 1512 Acked 2511 Read 6 New Done Done true 1513 Acked 2512 Read 4 New Done Done false 1514 Acked 2513 Write 9 New Done Done false 1515 Acked 2514 Read 1 New Done Done false 1516 Acked 2515 Read 10 New Done Done false 1517 Acked 2516 Flush 0 New Done Done false 1518 Acked 2517 Write 9 New Done Done false 1519 Acked 2518 Write 2 New Done Done false 1520 Acked 2519 Read 3 New Done Done false 1521 Acked 2520 Read 3 New Done Done false 1522 Acked 2521 Write 3 New Done Done false 1523 Acked 2522 Flush 0 New Done Done false 1524 Acked 2523 Flush 0 New Done Done false 1525 Acked 2524 Write 3 New Done Done false 1526 Acked 2525 Read 10 New Done Done false 1527 Acked 2526 Read 6 New Done Done false 1528 Acked 2527 Write 5 New Done Done false 1529 Acked 2528 Read 4 New Done Done false 1530 Acked 2529 Write 9 New Done Done false 1531 Acked 2530 Write 9 New Done Done false 1532 Acked 2531 Write 4 New Done Done false 1533 Acked 2532 Read 9 New Done Done false 1534 Acked 2533 Flush 0 New Done Done false 1535 Acked 2534 Write 4 New Done Done false 1536 Acked 2535 Read 3 New Done Done false 1537 Acked 2536 Write 9 New Done Done false 1538 Acked 2537 Read 3 New Done Done false 1539 Acked 2538 Read 3 New Done Done false 1540 Acked 2539 Read 8 New Done Done false 1541 Acked 2540 Flush 0 New Done Done false 1542 Acked 2541 Flush 0 New Done Done false 1543 Acked 2542 Write 9 New Done Done false 1544 Acked 2543 Write 4 New Done Done false 1545 Acked 2544 Flush 0 New Done Done false 1546 Acked 2545 Write 5 New Done Done false 1547 Acked 2546 Read 10 New Done Done false 1548 Acked 2547 Read 10 New Done Done false 1549 Acked 2548 Write 6 New Done Done false 1550 Acked 2549 Write 1 New Done Done false 1551 Acked 2550 Read 6 New Done Done false 1552 Acked 2551 Write 1 New Done Done false 1553 Acked 2552 Read 6 New Done Done false 1554 Acked 2553 Write 10 New Done Done false 1555 Acked 2554 Read 3 New Done Done false 1556 Acked 2555 Read 6 New Done Done false 1557 Acked 2556 Read 2 New Done Done false 1558 Acked 2557 Write 6 New Done Done false 1559 Acked 2558 Read 9 New Done Done false 1560 Acked 2559 Read 9 New Done Done false 1561 Acked 2560 Write 6 New Done Done false 1562 Acked 2561 Read 5 New Done Done false 1563 Acked 2562 Read 2 New Done Done false 1564 Acked 2563 Write 10 New Done Done false 1565 Acked 2564 Write 7 New Done Done false 1566 Acked 2565 Read 2 New Done Done false 1567 Acked 2566 Write 7 New Done Done false 1568 Acked 2567 Read 1 New Done Done false 1569 Acked 2568 Read 4 New Done Done false 1570 Acked 2569 Write 9 New Done Done false 1571 Acked 2570 Write 4 New Done Done false 1572 Acked 2571 Read 8 New Done Done false 1573 Acked 2572 Flush 0 New Done Done false 1574 Acked 2573 Read 10 New Done Done false 1575 Acked 2574 Read 6 New Done Done false 1576 Acked 2575 Write 10 New Done Done false 1577 Acked 2576 Write 7 New Done Done false 1578 Acked 2577 Write 9 New Done Done false 1579 Acked 2578 Flush 0 New Done Done false 1580 Acked 2579 Read 4 New Done Done false 1581 Acked 2580 Flush 0 New Done Done false 1582 Acked 2581 Write 5 New Done Done false 1583 Acked 2582 Flush 0 New Done Done false 1584 Acked 2583 Read 7 New Done Done false 1585 Acked 2584 Read 1 New Done Done false 1586 Acked 2585 Flush 0 New Done Done false 1587 Acked 2586 Write 4 New Done Done false 1588 Acked 2587 Write 1 New Done Done false 1589 Acked 2588 Read 3 New Done Done false 1590 Acked 2589 Read 1 New Done Done false 1591 Acked 2590 Write 9 New Done Done false 1592 Acked 2591 Write 9 New Done Done false 1593 Acked 2592 Write 9 New Done Done false 1594 Acked 2593 Write 6 New Done Done false 1595 Acked 2594 Write 10 New Done Done false 1596 Acked 2595 Write 7 New Done Done false 1597 Acked 2596 Write 6 New Done Done false 1598 Acked 2597 Write 1 New Done Done false 1599 Acked 2598 Write 1 New Done Done false 1600 Acked 2599 Write 5 New Done Done false 1601 Acked 2600 Write 7 New Done Done false 1602 Acked 2601 Write 9 New Done Done false 1603 Acked 2602 Flush 0 New Done Done false 1604 Acked 2603 Read 9 New Done Done false 1605 Acked 2604 Read 2 New Done Done false 1606 Acked 2605 Read 3 New Done Done false 1607 Acked 2606 Write 2 New Done Done false 1608 Acked 2607 Read 3 New Done Done false 1609 Acked 2608 Read 8 New Done Done false 1610 Acked 2609 Read 2 New Done Done false 1611 Acked 2610 Read 7 New Done Done false 1612 Acked 2611 Read 7 New Done Done false 1613 Acked 2612 Read 3 New Done Done false 1614 Acked 2613 Read 6 New Done Done false 1615 Acked 2614 Flush 0 New Done Done false 1616 Acked 2615 Read 6 New Done Done false 1617 Acked 2616 Write 6 New Done Done false 1618 Acked 2617 Write 5 New Done Done false 1619 Acked 2618 Read 10 New Done Done false 1620 Acked 2619 Flush 0 New Done Done false 1621 Acked 2620 Read 5 New Done Done false 1622 Acked 2621 Flush 0 New Done Done false 1623 Acked 2622 Write 9 New Done Done false 1624 Acked 2623 Read 7 New Done Done false 1625 Acked 2624 Write 3 New Done Done false 1626 Acked 2625 Read 10 New Done Done false 1627 Acked 2626 Read 4 New Done Done false 1628 Acked 2627 Read 9 New Done Done false 1629 Acked 2628 Write 9 New Done Done false 1630 Acked 2629 Read 10 New Done Done false 1631 Acked 2630 Read 6 New Done Done false 1632 Acked 2631 Write 9 New Done Done false 1633 Acked 2632 Write 2 New Done Done false 1634 Acked 2633 Flush 0 New Done Done false 1635 Acked 2634 Read 1 New Done Done false 1636 Acked 2635 Read 3 New Done Done false 1637 Acked 2636 Flush 0 New Done Done false 1638 Acked 2637 Write 4 New Done Done false 1639 Acked 2638 Write 2 New Done Done false 1640 Acked 2639 Read 5 New Done Done false 1641 Acked 2640 Flush 0 New Done Done false 1642 Acked 2641 Read 1 New Done Done false 1643 Acked 2642 Read 1 New Done Done false 1644 Acked 2643 Read 1 New Done Done false 1645 Acked 2644 Write 3 New Done Done false 1646 Acked 2645 Write 1 New Done Done false 1647 Acked 2646 Read 5 New Done Done false 1648 Acked 2647 Flush 0 New Done Done false 1649 Acked 2648 Read 8 New Done Done false 1650 Acked 2649 Write 10 New Done Done false 1651 Acked 2650 Read 10 New Done Done false 1652 Acked 2651 Read 10 New Done Done false 1653 Acked 2652 Read 5 New Done Done false 1654 Acked 2653 Read 3 New Done Done false 1655 Acked 2654 Read 7 New Done Done false 1656 Acked 2655 Write 3 New Done Done false 1657 Acked 2656 Read 10 New Done Done false 1658 Acked 2657 Read 2 New Done Done false 1659 Acked 2658 Flush 0 New Done Done false 1660 Acked 2659 Write 5 New Done Done false 1661 Acked 2660 Read 3 New Done Done false 1662 Acked 2661 Read 1 New Done Done false 1663 Acked 2662 Read 1 New Done Done false 1664 Acked 2663 Flush 0 New Done Done false 1665 Acked 2664 Read 5 New Done Done false 1666 Acked 2665 Write 2 New Done Done false 1667 Acked 2666 Flush 0 New Done Done false 1668 Acked 2667 Write 10 New Done Done false 1669 Acked 2668 Read 4 New Done Done false 1670 Acked 2669 Read 9 New Done Done false 1671 Acked 2670 Flush 0 New Done Done false 1672 Acked 2671 Write 3 New Done Done false 1673 Acked 2672 Read 5 New Done Done false 1674 Acked 2673 Write 2 New Done Done false 1675 Acked 2674 Write 8 New Done Done false 1676 Acked 2675 Flush 0 New Done Done false 1677 Acked 2676 Read 1 New Done Done false 1678 Acked 2677 Read 2 New Done Done false 1679 Acked 2678 Write 2 New Done Done false 1680 Acked 2679 Read 6 New Done Done false 1681 Acked 2680 Read 5 New Done Done false 1682 Acked 2681 Flush 0 New Done Done false 1683 Acked 2682 Write 3 New Done Done false 1684 Acked 2683 Write 6 New Done Done false 1685 Acked 2684 Write 10 New Done Done false 1686 Acked 2685 Read 4 New Done Done false 1687 Acked 2686 Flush 0 New Done Done false 1688 Acked 2687 Write 6 New Done Done false 1689 Acked 2688 Read 5 New Done Done false 1690 Acked 2689 Write 5 New Done Done false 1691 Acked 2690 Read 1 New Done Done false 1692 Acked 2691 Read 7 New Done Done false 1693 Acked 2692 Write 6 New Done Done false 1694 Acked 2693 Write 4 New Done Done false 1695 Acked 2694 Flush 0 New Done Done false 1696 Acked 2695 Write 7 New Done Done false 1697 Acked 2696 Read 2 New Done Done false 1698 Acked 2697 Write 2 New Done Done false 1699 Acked 2698 Flush 0 New Done Done false 1700 Acked 2699 Read 4 New Done Done false 1701 Acked 2700 Write 5 New Done Done false 1702 Acked 2701 Read 6 New Done Done false 1703 Acked 2702 Write 9 New Done Done false 1704 Acked 2703 Read 9 New Done Done false 1705 Acked 2704 Write 6 New Done Done false 1706 Acked 2705 Read 9 New Done Done false 1707 Acked 2706 Read 6 New Done Done false 1708 Acked 2707 Write 3 New Done Done false 1709 Acked 2708 Write 6 New Done Done false 1710 Acked 2709 Read 4 New Done Done false 1711 Acked 2710 Read 8 New Done Done false 1712 Acked 2711 Read 3 New Done Done false 1713 Acked 2712 Read 7 New Done Done false 1714 Acked 2713 Flush 0 New Done Done false 1715 Acked 2714 Read 4 New Done Done false 1716 Acked 2715 Flush 0 New Done Done false 1717 Acked 2716 Write 2 New Done Done false 1718 Acked 2717 Write 1 New Done Done false 1719 Acked 2718 Read 9 New Done Done false 1720 Acked 2719 Write 4 New Done Done false 1721 Acked 2720 Write 9 New Done Done false 1722 Acked 2721 Write 3 New Done Done false 1723 Acked 2722 Write 1 New Done Done false 1724 Acked 2723 Read 5 New Done Done false 1725 Acked 2724 Write 1 New Done Done false 1726 Acked 2725 Flush 0 New Done Done false 1727 Acked 2726 Read 7 New Done Done false 1728 Acked 2727 Read 2 New Done Done false 1729 Acked 2728 Write 10 New Done Done false 1730 Acked 2729 Flush 0 New Done Done false 1731 Acked 2730 Read 7 New Done Done false 1732 Acked 2731 Read 4 New Done Done false 1733 Acked 2732 Write 7 New Done Done false 1734 Acked 2733 Flush 0 New Done Done false 1735 Acked 2734 Write 6 New Done Done false 1736 Acked 2735 Read 6 New Done Done false 1737 Acked 2736 Write 9 New Done Done false 1738 Acked 2737 Write 5 New Done Done false 1739 Acked 2738 Flush 0 New Done Done false 1740 Acked 2739 Read 7 New Done Done false 1741 Acked 2740 Flush 0 New Done Done false 1742 Acked 2741 Read 3 New Done Done false 1743 Acked 2742 Read 6 New Done Done false 1744 Acked 2743 Write 2 New Done Done false 1745 Acked 2744 Read 6 New Done Done false 1746 Acked 2745 Read 2 New Done Done false 1747 Acked 2746 Write 6 New Done Done false 1748 Acked 2747 Read 1 New Done Done false 1749 Acked 2748 Write 6 New Done Done false 1750 Acked 2749 Read 2 New Done Done false 1751 Acked 2750 Read 1 New Done Done false 1752 Acked 2751 Write 9 New Done Done false 1753 Acked 2752 Write 6 New Done Done false 1754 Acked 2753 Read 10 New Done Done false 1755 Acked 2754 Write 2 New Done Done false 1756 Acked 2755 Flush 0 New Done Done false 1757 Acked 2756 Read 4 New Done Done false 1758 Acked 2757 Write 2 New Done Done false 1759 Acked 2758 Read 9 New Done Done false 1760 Acked 2759 Flush 0 New Done Done false 1761 Acked 2760 Write 10 New Done Done false 1762 Acked 2761 Write 8 New Done Done false 1763 Acked 2762 Flush 0 New Done Done false 1764 Acked 2763 Flush 0 New Done Done false 1765 Acked 2764 Read 3 New Done Done false 1766 Acked 2765 Write 8 New Done Done false 1767 Acked 2766 Read 10 New Done Done false 1768 Acked 2767 Read 9 New Done Done false 1769 Acked 2768 Read 5 New Done Done false 1770 Acked 2769 Write 9 New Done Done false 1771 Acked 2770 Read 9 New Done Done false 1772 Acked 2771 Read 10 New Done Done false 1773 Acked 2772 Read 3 New Done Done false 1774 Acked 2773 Write 9 New Done Done false 1775 Acked 2774 Flush 0 New Done Done false 1776 Acked 2775 Read 3 New Done Done false 1777 Acked 2776 Read 5 New Done Done false 1778 Acked 2777 Read 9 New Done Done false 1779 Acked 2778 Read 1 New Done Done false 1780 Acked 2779 Read 9 New Done Done false 1781 Acked 2780 Read 10 New Done Done false 1782 Acked 2781 Write 10 New Done Done false 1783 Acked 2782 Read 2 New Done Done false 1784 Acked 2783 Flush 0 New Done Done false 1785 Acked 2784 Read 3 New Done Done false 1786 Acked 2785 Write 1 New Done Done false 1787 Acked 2786 Read 4 New Done Done false 1788 Acked 2787 Write 6 New Done Done false 1789 Acked 2788 Read 5 New Done Done false 1790 Acked 2789 Read 2 New Done Done false 1791 Acked 2790 Read 2 New Done Done false 1792 Acked 2791 Read 4 New Done Done false 1793 Acked 2792 Read 2 New Done Done false 1794 Acked 2793 Write 5 New Done Done false 1795 Acked 2794 Read 4 New Done Done false 1796 Acked 2795 Write 5 New Done Done false 1797 Acked 2796 Read 9 New Done Done false 1798 Acked 2797 Write 3 New Done Done false 1799 Acked 2798 Write 1 New Done Done false 1800 Acked 2799 Read 6 New Done Done false 1801 Acked 2800 Write 4 New Done Done false 1802 Acked 2801 Flush 0 New Done Done false 1803 Acked 2802 Write 7 New Done Done false 1804 Acked 2803 Read 10 New Done Done false 1805 Acked 2804 Read 3 New Done Done false 1806 Acked 2805 Read 4 New Done Done false 1807 Acked 2806 Read 5 New Done Done false 1808 Acked 2807 Read 4 New Done Done false 1809 Acked 2808 Read 6 New Done Done false 1810 Acked 2809 Flush 0 New Done Done false 1811 Acked 2810 Read 7 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 2509 2809 2809 Downstairs last five completed: 2509 2508 2507 2506 2505 Upstairs last five completed: 1811 1810 1809 1808 1807 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:36.03189632Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:36.031966306Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:36.031994888Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:36.032021762Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:36.032042264Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:36.032076687Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:36.032334737Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:37.033161212Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:37.033232241Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:37.033260212Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:37.033284058Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:37.03331038Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:37.033340997Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:37.033368883Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:37.03362675Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1191) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.034606337Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.034759997Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.036763689Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.036814394Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.036838315Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 2509","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.037109506Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.037139332Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.037162884Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.037185253Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:38.0372076Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 2811 2810 2809 2808 2807 Upstairs last five completed: 1812 1811 1810 1809 1808 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(1) stop 1 Jan 02 22:31:39.943 INFO request completed, latency_us: 449, response_code: 204[8820] Got stop action so:false kr:false , uri: /stop/cid/1, method: POST, req_id: 76d46519-02dc-4cfb-9ce0-28f91feb0f22, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:39.945195293Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:39.945283808Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:31:39.945 {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:39.945329357Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/1, method: GET, req_id: 711ed933-56e8-4a0f-87d5-279af81d7126, remote_addr:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:39.945404529Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"[1] client re-new 0 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:39.945447325Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:39.945483783Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:40.947101469Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:40.947174481Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:40.94720271Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:40.947229435Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:40.947249745Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:40.947274579Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:40.947319434Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:41.949055846Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:41.949130357Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:41.949158666Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:41.94918284Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:41.949209163Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:41.94922956Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:41.949251905Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:41.949279789Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:42.950260246Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:42.950328774Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:42.950356784Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:42.950383172Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:42.950414626Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:42.950438194Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:42.950547669Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:31:43.947 INFO request completed, latency_us: 241, response_code: 200, uri: /state/cid/1, method: GET, req_id: d9e27b5e-f0c3-4106-a487-f8f3bb69f033, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 3113 len 20480 data: 2 2 3 2 2 002/300 Flush {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:43.951135948Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:43.951202037Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:43.951230411Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:43.951256954Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:43.951277488Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 2 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:43.951300048Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:43.951407122Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Write block 4003 len 12288 data: 2 2 2 004/300 Read block 2597 len 20480 005/300 Read block 91 len 36864 006/300 Read block 117 len 20480 007/300 Read block 4931 len 40960 008/300 Write block 3841 len 32768 data: 3 2 2 2 2 2 2 2 009/300 Read block 3715 len 32768 010/300 Write block 3106 len 8192 data: 2 2 011/300 Flush 012/300 Read block 1719 len 32768 013/300 Read block 523 len 8192 014/300 Read block 594 len 32768 015/300 Write block 313 len 4096 data: 5 016/300 Write block 325 len 40960 data: 3 2 2 2 2 2 2 3 3 3 017/300 Write block 4014 len 20480 data: 3 3 3 3 3 018/300 Read block 4602 len 24576 019/300 Read block 3005 len 20480 020/300 Read block 4328 len 28672 021/300 Write block 576 len 40960 data: 2 2 3 3 3 3 3 3 2 2 022/300 Write block 2845 len 20480 data: 3 3 3 3 2 023/300 Flush 024/300 Read block 4951 len 24576 025/300 Flush 026/300 Write block 2296 len 36864 data: 2 2 2 2 2 2 2 2 2 027/300 Read block 1083 len 16384 028/300 Read block 1993 len 40960 029/300 Write block 4986 len 28672 data: 4 3 3 3 2 2 2 030/300 Read block 869 len 36864 031/300 Read block 4062 len 4096 032/300 Read block 2923 len 16384 033/300 Write block 3862 len 40960 data: 4 4 4 3 3 2 4 4 4 4 034/300 Write block 134 len 20480 data: 3 3 3 2 2 035/300 Read block 2464 len 36864 036/300 Write block 4866 len 28672 data: 3 3 3 5 4 4 4 037/300 Write block 3474 len 8192 data: 2 2 038/300 Read block 2180 len 40960 039/300 Write block 849 len 20480 data: 4 4 4 4 3 040/300 Read block 48 len 20480 041/300 Read block 2603 len 8192 042/300 Write block 158 len 16384 data: 3 3 4 3 043/300 Write block 4019 len 32768 data: 2 2 2 2 3 3 3 3 044/300 Read block 4632 len 16384 045/300 Write block 4731 len 40960 data: 4 5 6 6 6 6 6 6 4 4 046/300 Write block 1765 len 36864 data: 2 2 2 3 3 3 3 4 4 047/300 Read block 3407 len 36864 048/300 Read block 1734 len 4096 049/300 Flush 050/300 Read block 2267 len 40960 051/300 Read block 1612 len 24576 052/300 Write block 2936 len 4096 data: 3 053/300 Write block 2072 len 8192 data: 2 2 054/300 Read block 769 len 28672 055/300 Write block 2857 len 28672 data: 3 3 3 3 2 2 3 056/300 Write block 3343 len 12288 data: 3 4 4 057/300 Read block 3139 len 28672 058/300 Read block 2571 len 24576 059/300 Flush 060/300 Read block 163 len 20480 061/300 Read block 3682 len 40960 062/300 Read block 317 len 32768 063/300 Read block 4299 len 28672 064/300 Read block 2733 len 36864 065/300 Flush 066/300 Read block 1940 len 16384 067/300 Read block 1669 len 20480 068/300 Read block 4546 len 28672 069/300 Write block 2209 len 20480 data: 4 4 4 3 3 070/300 Write block 1253 len 40960 data: 2 2 2 3 4 4 3 3 3 2 071/300 Write block 2410 len 36864 data: 3 3 4 5 5 4 5 5 6 072/300 Flush 073/300 Read block 1155 len 12288 074/300 Read block 836 len 16384 075/300 Read block 4709 len 20480 076/300 Read block 1633 len 24576 077/300 Read block 901 len 20480 078/300 Read block 4926 len 20480 079/300 Flush 080/300 Read block 4713 len 28672 081/300 Read block 3036 len 36864 082/300 Read block 924 len 40960 083/300 Write block 4307 len 32768 data: 2 2 2 2 2 2 2 2 084/300 Read block 3193 len 24576 085/300 Read block 2505 len 16384 086/300 Write block 4223 len 40960 data: 3 3 3 3 3 3 3 3 3 3 087/300 Read block 2943 len 4096 088/300 Flush 089/300 Read block 1082 len 12288 090/300 Write block 2821 len 28672 data: 4 4 4 4 5 5 5 091/300 Write block 731 len 24576 data: 3 3 2 2 2 2 092/300 Write block 1639 len 12288 data: 2 2 2 093/300 Read block 3310 len 40960 094/300 Read block 4021 len 28672 095/300 Flush 096/300 Write block 2912 len 8192 data: 2 3 097/300 Flush 098/300 Write block 1462 len 32768 data: 2 2 2 2 2 2 2 2 099/300 Write block 4753 len 20480 data: 2 2 2 2 2 100/300 Read block 4849 len 32768 101/300 Read block 3349 len 36864 102/300 Write block 3813 len 16384 data: 2 2 2 2 103/300 Read block 3567 len 24576 104/300 Write block 4162 len 32768 data: 2 2 3 3 3 3 3 3 105/300 Write block 3013 len 40960 data: 3 3 3 3 3 4 3 3 3 3 106/300 Flush 107/300 Read block 2734 len 16384 108/300 Read block 4512 len 28672 109/300 Write block 2620 len 40960 data: 2 2 2 2 2 2 2 3 3 2 110/300 Write block 3138 len 28672 data: 5 4 4 4 3 3 2 111/300 Read block 3211 len 40960 112/300 Read block 1769 len 32768 113/300 Read block 3662 len 12288 114/300 Read block 4765 len 36864 115/300 Write block 607 len 32768 data: 2 2 2 2 2 2 2 3 116/300 Write block 2659 len 4096 data: 3 117/300 Read block 3682 len 24576 118/300 Read block 4356 len 20480 119/300 Flush 120/300 Write block 853 len 40960 data: 4 3 2 2 2 2 2 2 2 2 121/300 Write block 4771 len 8192 data: 4 4 122/300 Read block 1037 len 28672 123/300 Read block 4371 len 12288 124/300 Read block 1049 len 20480 125/300 Flush 126/300 Read block 1423 len 16384 127/300 Write block 2487 len 28672 data: 3 4 4 3 2 2 2 128/300 Read block 956 len 12288 129/300 Read block 4374 len 12288 130/300 Read block 4049 len 20480 131/300 Read block 2190 len 36864 132/300 Write block 4887 len 32768 data: 5 3 2 2 2 2 2 2 133/300 Flush 134/300 Read block 2777 len 24576 135/300 Read block 4983 len 12288 136/300 Write block 3099 len 32768 data: 3 3 3 3 2 2 2 3 137/300 Write block 318 len 24576 data: 3 3 2 2 2 3 138/300 Read block 1308 len 16384 139/300 Read block 352 len 36864 140/300 Read block 2909 len 8192 141/300 Read block 2685 len 24576 142/300 Flush 143/300 Write block 3077 len 40960 data: 2 2 2 3 3 4 4 3 3 3 144/300 Read block 4298 len 40960 145/300 Write block 2234 len 36864 data: 2 2 2 2 2 2 3 2 2 146/300 Read block 4524 len 16384 147/300 Read block 168 len 36864 148/300 Read block 2320 len 4096 149/300 Read block 2960 len 12288 150/300 Read block 3946 len 8192 151/300 Write block 4082 len 40960 data: 3 3 3 3 3 3 3 2 2 2 152/300 Write block 1263 len 12288 data: 2 2 2 153/300 Read block 3462 len 8192 154/300 Read block 3987 len 36864 155/300 Read block 4628 len 36864 156/300 Read block 2095 len 40960 157/300 Read block 3324 len 4096 158/300 Read block 3585 len 32768 159/300 Read block 4484 len 32768 160/300 Write block 771 len 24576 data: 2 3 3 3 2 2 161/300 Read block 950 len 24576 162/300 Write block 3014 len 40960 data: 4 4 4 4 5 4 4 4 4 2 163/300 Write block 4293 len 28672 data: 2 2 2 2 2 2 2 164/300 Write block 283 len 20480 data: 3 3 3 3 3 165/300 Flush 166/300 Flush 167/300 Write block 525 len 24576 data: 2 2 2 2 2 2 168/300 Write block 2729 len 36864 data: 3 2 2 2 3 3 3 3 3 169/300 Write block 3339 len 40960 data: 4 5 5 3 4 5 5 4 4 4 170/300 Write block 864 len 20480 data: 2 2 3 3 3 171/300 Read block 80 len 12288 172/300 Read block 3870 len 12288 173/300 Flush 174/300 Read block 4184 len 36864 175/300 Read block 2475 len 20480 176/300 Write block 2641 len 24576 data: 2 2 2 3 3 3 177/300 Write block 4730 len 36864 data: 4 5 6 7 7 7 7 7 7 178/300 Read block 371 len 20480 179/300 Read block 1772 len 16384 180/300 Write block 2233 len 32768 data: 2 3 3 3 3 3 3 4 181/300 Flush 182/300 Write block 128 len 16384 data: 3 4 3 3 183/300 Read block 3112 len 36864 184/300 Write block 3258 len 36864 data: 3 4 4 3 4 4 4 4 4 185/300 Read block 607 len 4096 186/300 Read block 4365 len 24576 187/300 Write block 1350 len 24576 data: 6 4 4 4 3 4 188/300 Read block 4248 len 36864 189/300 Read block 1281 len 32768 190/300 Read block 2256 len 12288 191/300 Write block 1644 len 8192 data: 2 2 192/300 Write block 426 len 28672 data: 2 2 2 2 2 3 3 193/300 Read block 1059 len 40960 194/300 Read block 2067 len 24576 195/300 Write block 1069 len 36864 data: 3 4 3 3 3 2 2 2 2 196/300 Read block 4827 len 16384 197/300 Write block 3926 len 40960 data: 4 4 3 3 4 4 4 3 2 2 198/300 Read block 238 len 32768 199/300 Read block 4087 len 40960 200/300 Read block 351 len 12288 201/300 Read block 2303 len 36864 202/300 Write block 756 len 24576 data: 3 3 3 3 2 2 203/300 Read block 3424 len 4096 204/300 Write block 1488 len 36864 data: 2 2 2 2 2 2 2 3 2 205/300 Write block 2502 len 40960 data: 2 2 2 3 4 4 4 3 3 3 206/300 Write block 4233 len 40960 data: 3 3 2 2 2 2 2 2 2 2 207/300 Read block 1036 len 32768 208/300 Write block 338 len 36864 data: 2 3 3 4 4 3 2 2 2 209/300 Read block 2562 len 20480 210/300 Read block 2449 len 4096 211/300 Write block 1775 len 28672 data: 5 4 3 3 3 3 3 212/300 Read block 2456 len 20480 213/300 Write block 3946 len 8192 data: 2 3 214/300 Write block 2797 len 28672 data: 2 2 4 4 4 4 4 215/300 Read block 4648 len 28672 216/300 Read block 2940 len 12288 217/300 Write block 3823 len 4096 data: 2 218/300 Read block 1384 len 8192 219/300 Flush 220/300 Flush 221/300 Read block 1856 len 16384 222/300 Write block 1653 len 40960 data: 3 2 5 5 5 4 4 4 4 3 223/300 Read block 2969 len 4096 224/300 Flush 225/300 Read block 4746 len 20480 226/300 Read block 3072 len 40960 227/300 Write block 148 len 32768 data: 3 3 3 2 3 3 3 3 228/300 Read block 1744 len 4096 229/300 Write block 3077 len 32768 data: 3 3 3 4 4 5 5 4 230/300 Read block 3047 len 40960 231/300 Read block 3097 len 4096 232/300 Read block 2976 len 28672 233/300 Write block 2662 len 16384 data: 2 2 2 2 234/300 Flush 235/300 Write block 399 len 4096 data: 3 236/300 Read block 2676 len 32768 237/300 Write block 1946 len 20480 data: 3 3 3 3 2 238/300 Read block 22 len 28672 239/300 Read block 774 len 12288 240/300 Flush 241/300 Write block 2948 len 24576 data: 5 5 5 5 4 4 242/300 Read block 1667 len 36864 243/300 Write block 83 len 24576 data: 2 2 2 2 2 2 244/300 Read block 2948 len 40960 245/300 Write block 1510 len 8192 data: 4 3 246/300 Read block 575 len 24576 247/300 Read block 3159 len 8192 248/300 Flush 249/300 Write block 2781 len 20480 data: 2 2 2 2 2 250/300 Read block 3050 len 8192 251/300 Read block 3717 len 40960 252/300 Flush 253/300 Flush 254/300 Read block 3891 len 36864 255/300 Read block 2516 len 20480 256/300 Read block 2910 len 4096 257/300 Write block 860 len 32768 data: 3 3 3 2 3 3 4 4 258/300 Read block 4050 len 24576 259/300 Write block 1608 len 40960 data: 2 2 2 2 2 2 2 2 2 2 260/300 Write block 2775 len 24576 data: 3 3 3 2 2 3 261/300 Write block 1298 len 12288 data: 2 2 2 262/300 Flush 263/300 Write block 406 len 8192 data: 2 2 264/300 Write block 4541 len 16384 data: 3 3 2 2 265/300 Read block 4365 len 4096 266/300 Write block 3616 len 32768 data: 3 3 3 3 4 4 5 6 267/300 Flush 268/300 Read block 1970 len 40960 269/300 Read block 861 len 4096 270/300 Read block 1769 len 40960 271/300 Write block 3923 len 24576 data: 4 4 4 5 5 4 272/300 Read block 2329 len 8192 273/300 Write block 136 len 12288 data: 4 3 3 274/300 Read block 4139 len 40960 275/300 Read block 1238 len 4096 276/300 Read block 3741 len 20480 277/300 Read block 1921 len 32768 278/300 Write block 4877 len 36864 data: 3 2 3 3 4 4 4 4 5 279/300 Read block 609 len 36864 280/300 Read block 3515 len 8192 281/300 Write block 1398 len 32768 data: 2 2 2 2 2 2 2 2 282/300 Read block 4715 len 4096 283/300 Read block 1529 len 24576 284/300 Write block 4320 len 16384 data: 4 4 4 4 285/300 Read block 3223 len 32768 286/300 Write block 3792 len 32768 data: 4 3 3 2 2 2 3 3 287/300 Read block 1081 len 20480 288/300 Read block 4919 len 20480 289/300 Read block 4441 len 36864 290/300 Flush {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:44.953111812Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:44.953269121Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:44.953321323Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:44.953374123Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:44.953414984Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 292 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:44.953441805Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:44.953464191Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 291/300 Write block 1001 len 8192 data: 3 3 292/300 Read block 2597 len 4096 293/300 Write block 2045 len 20480 data: 3 3 5 6 4 294/300 Write block 3637 len 20480 data: 3 3 2 2 2 295/300 Write block 4556 len 36864 data: 2 2 2 2 2 2 2 2 2 296/300 Write block 3054 len 40960 data: 4 4 4 4 3 3 2 2 3 2 297/300 Write block 4675 len 16384 data: 3 2 2 2 298/300 Read block 2156 len 16384 299/300 Write block 4003 len 4096 data: 3 300/300 Write block 3074 len 16384 data: 2 2 2 4 Main task has work to do, go find it got dsc Start(1) start 1 Jan 02 22:31:44.965 INFO request completed, latency_us: 301, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9eae4b04-5515-4819-a546-9004dce20729, remote_addr: 127.0.0.Replay: started 1, returned:Ok(()) 1:37434, local_addr: 127.0.0.1:9998 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1813 Acked 2812 Write 5 Done New Done true 1814 Acked 2813 Flush 0 Done New Done true 1815 Acked 2814 Write 3 Done New Done true 1816 Acked 2815 Read 5 Done New Done true 1817 Acked 2816 Read 9 Done New Done true 1818 Acked 2817 Read 5 Done New Done true 1819 Acked 2818 Read 10 Done New Done true 1820 Acked 2819 Write 8 Done New Done true 1821 Acked 2820 Read 8 Done New Done true 1822 Acked 2821 Write 2 Done New Done true 1823 Acked 2822 Flush 0 Done New Done true 1824 Acked 2823 Read 8 Done New Done true 1825 Acked 2824 Read 2 Done New Done true 1826 Acked 2825 Read 8 Done New Done true 1827 Acked 2826 Write 1 Done New Done true 1828 Acked 2827 Write 10 Done New Done true 1829 Acked 2828 Write 5 Done New Done true 1830 Acked 2829 Read 6 Done New Done true 1831 Acked 2830 Read 5 Done New Done true 1832 Acked 2831 Read 7 Done New Done true 1833 Acked 2832 Write 10 Done New Done true 1834 Acked 2833 Write 5 Done New Done true 1835 Acked 2834 Flush 0 Done New Done true 1836 Acked 2835 Read 6 Done New Done true 1837 Acked 2836 Flush 0 Done New Done true 1838 Acked 2837 Write 9 Done New Done true 1839 Acked 2838 Read 4 Done New Done true 1840 Acked 2839 Read 10 Done New Done true 1841 Acked 2840 Write 7 Done New Done true 1842 Acked 2841 Read 9 Done New Done true 1843 Acked 2842 Read 1 Done New Done true 1844 Acked 2843 Read 4 Done New Done true 1845 Acked 2844 Write 10 Done New Done true 1846 Acked 2845 Write 5 Done New Done true 1847 Acked 2846 Read 9 Done New Done true 1848 Acked 2847 Write 7 Done New Done true 1849 Acked 2848 Write 2 Done New Done true 1850 Acked 2849 Read 10 Done New Done true 1851 Acked 2850 Write 5 Done New Done true 1852 Acked 2851 Read 5 Done New Done true 1853 Acked 2852 Read 2 Done New Done true 1854 Acked 2853 Write 4 Done New Done true 1855 Acked 2854 Write 8 Done New Done true 1856 Acked 2855 Read 4 Done New Done true 1857 Acked 2856 Write 10 Done New Done true 1858 Acked 2857 Write 9 Done New Done true 1859 Acked 2858 Read 9 Done New Done true 1860 Acked 2859 Read 1 Done New Done true 1861 Acked 2860 Flush 0 Done New Done true 1862 Acked 2861 Read 10 Done New Done true 1863 Acked 2862 Read 6 Done New Done true 1864 Acked 2863 Write 1 Done New Done true 1865 Acked 2864 Write 2 Done New Done true 1866 Acked 2865 Read 7 Done New Done true 1867 Acked 2866 Write 7 Done New Done true 1868 Acked 2867 Write 3 Done New Done true 1869 Acked 2868 Read 7 Done New Done true 1870 Acked 2869 Read 6 Done New Done true 1871 Acked 2870 Flush 0 Done New Done true 1872 Acked 2871 Read 5 Done New Done true 1873 Acked 2872 Read 10 Done New Done true 1874 Acked 2873 Read 8 Done New Done true 1875 Acked 2874 Read 7 Done New Done true 1876 Acked 2875 Read 9 Done New Done true 1877 Acked 2876 Flush 0 Done New Done true 1878 Acked 2877 Read 4 Done New Done true 1879 Acked 2878 Read 5 Done New Done true 1880 Acked 2879 Read 7 Done New Done true 1881 Acked 2880 Write 5 Done New Done true 1882 Acked 2881 Write 10 Done New Done true 1883 Acked 2882 Write 9 Done New Done true 1884 Acked 2883 Flush 0 Done New Done true 1885 Acked 2884 Read 3 Done New Done true 1886 Acked 2885 Read 4 Done New Done true 1887 Acked 2886 Read 5 Done New Done true 1888 Acked 2887 Read 6 Done New Done true 1889 Acked 2888 Read 5 Done New Done true 1890 Acked 2889 Read 5 Done New Done true 1891 Acked 2890 Flush 0 Done New Done true 1892 Acked 2891 Read 7 Done New Done true 1893 Acked 2892 Read 9 Done New Done true 1894 Acked 2893 Read 10 Done New Done true 1895 Acked 2894 Write 8 Done New Done true 1896 Acked 2895 Read 6 Done New Done true 1897 Acked 2896 Read 4 Done New Done true 1898 Acked 2897 Write 10 Done New Done true 1899 Acked 2898 Read 1 Done New Done true 1900 Acked 2899 Flush 0 Done New Done true 1901 Acked 2900 Read 3 Done New Done true 1902 Acked 2901 Write 7 Done New Done true 1903 Acked 2902 Write 6 Done New Done true 1904 Acked 2903 Write 3 Done New Done true 1905 Acked 2904 Read 10 Done New Done true 1906 Acked 2905 Read 7 Done New Done true 1907 Acked 2906 Flush 0 Done New Done true 1908 Acked 2907 Write 2 Done New Done true 1909 Acked 2908 Flush 0 Done New Done true 1910 Acked 2909 Write 8 Done New Done true 1911 Acked 2910 Write 5 Done New Done true 1912 Acked 2911 Read 8 Done New Done true 1913 Acked 2912 Read 9 Done New Done true 1914 Acked 2913 Write 4 Done New Done true 1915 Acked 2914 Read 6 Done New Done true 1916 Acked 2915 Write 8 Done New Done true 1917 Acked 2916 Write 10 Done New Done true 1918 Acked 2917 Flush 0 Done New Done true 1919 Acked 2918 Read 4 Done New Done true 1920 Acked 2919 Read 7 Done New Done true 1921 Acked 2920 Write 10 Done New Done true 1922 Acked 2921 Write 7 Done New Done true 1923 Acked 2922 Read 10 Done New Done true 1924 Acked 2923 Read 8 Done New Done true 1925 Acked 2924 Read 3 Done New Done true 1926 Acked 2925 Read 9 Done New Done true 1927 Acked 2926 Write 8 Done New Done true 1928 Acked 2927 Write 1 Done New Done true 1929 Acked 2928 Read 6 Done New Done true 1930 Acked 2929 Read 5 Done New Done true 1931 Acked 2930 Flush 0 Done New Done true 1932 Acked 2931 Write 10 Done New Done true 1933 Acked 2932 Write 2 Done New Done true 1934 Acked 2933 Read 7 Done New Done true 1935 Acked 2934 Read 3 Done New Done true 1936 Acked 2935 Read 5 Done New Done true 1937 Acked 2936 Flush 0 Done New Done true 1938 Acked 2937 Read 4 Done New Done true 1939 Acked 2938 Write 7 Done New Done true 1940 Acked 2939 Read 3 Done New Done true 1941 Acked 2940 Read 3 Done New Done true 1942 Acked 2941 Read 5 Done New Done true 1943 Acked 2942 Read 9 Done New Done true 1944 Acked 2943 Write 8 Done New Done true 1945 Acked 2944 Flush 0 Done New Done true 1946 Acked 2945 Read 6 Done New Done true 1947 Acked 2946 Read 3 Done New Done true 1948 Acked 2947 Write 8 Done New Done true 1949 Acked 2948 Write 6 Done New Done true 1950 Acked 2949 Read 4 Done New Done true 1951 Acked 2950 Flush 0 Done New Done true 1952 Acked 2951 Read 9 Done New Done true 1953 Acked 2952 Read 2 Done New Done true 1954 Acked 2953 Read 6 Done New Done true 1955 Acked 2954 Flush 0 Done New Done true 1956 Acked 2955 Write 10 Done New Done true 1957 Acked 2956 Read 10 Done New Done true 1958 Acked 2957 Write 9 Done New Done true 1959 Acked 2958 Read 4 Done New Done true 1960 Acked 2959 Read 9 Done New Done true 1961 Acked 2960 Read 1 Done New Done true 1962 Acked 2961 Read 3 Done New Done true 1963 Acked 2962 Read 2 Done New Done true 1964 Acked 2963 Write 10 Done New Done true 1965 Acked 2964 Write 3 Done New Done true 1966 Acked 2965 Read 2 Done New Done true 1967 Acked 2966 Read 9 Done New Done true 1968 Acked 2967 Read 9 Done New Done true 1969 Acked 2968 Read 10 Done New Done true 1970 Acked 2969 Read 1 Done New Done true 1971 Acked 2970 Read 8 Done New Done true 1972 Acked 2971 Read 8 Done New Done true 1973 Acked 2972 Write 6 Done New Done true 1974 Acked 2973 Read 6 Done New Done true 1975 Acked 2974 Write 10 Done New Done true 1976 Acked 2975 Write 7 Done New Done true 1977 Acked 2976 Write 5 Done New Done true 1978 Acked 2977 Flush 0 Done New Done true 1979 Acked 2978 Flush 0 Done New Done true 1980 Acked 2979 Write 6 Done New Done true 1981 Acked 2980 Write 9 Done New Done true 1982 Acked 2981 Write 10 Done New Done true 1983 Acked 2982 Write 5 Done New Done true 1984 Acked 2983 Read 3 Done New Done true 1985 Acked 2984 Read 3 Done New Done true 1986 Acked 2985 Flush 0 Done New Done true 1987 Acked 2986 Read 9 Done New Done true 1988 Acked 2987 Read 5 Done New Done true 1989 Acked 2988 Write 6 Done New Done true 1990 Acked 2989 Write 9 Done New Done true 1991 Acked 2990 Read 5 Done New Done true 1992 Acked 2991 Read 4 Done New Done true 1993 Acked 2992 Write 8 Done New Done true 1994 Acked 2993 Flush 0 Done New Done true 1995 Acked 2994 Write 4 Done New Done true 1996 Acked 2995 Read 9 Done New Done true 1997 Acked 2996 Write 9 Done New Done true 1998 Acked 2997 Read 1 Done New Done true 1999 Acked 2998 Read 6 Done New Done true 2000 Acked 2999 Write 6 Done New Done true 2001 Acked 3000 Read 9 Done New Done true 2002 Acked 3001 Read 8 Done New Done true 2003 Acked 3002 Read 3 Done New Done true 2004 Acked 3003 Write 2 Done New Done true 2005 Acked 3004 Write 7 Done New Done true 2006 Acked 3005 Read 10 Done New Done true 2007 Acked 3006 Read 6 Done New Done true 2008 Acked 3007 Write 9 Done New Done true 2009 Acked 3008 Read 4 Done New Done true 2010 Acked 3009 Write 10 Done New Done true 2011 Acked 3010 Read 8 Done New Done true 2012 Acked 3011 Read 10 Done New Done true 2013 Acked 3012 Read 3 Done New Done true 2014 Acked 3013 Read 9 Done New Done true 2015 Acked 3014 Write 6 Done New Done true 2016 Acked 3015 Read 1 Done New Done true 2017 Acked 3016 Write 9 Done New Done true 2018 Acked 3017 Write 10 Done New Done true 2019 Acked 3018 Write 10 Done New Done true 2020 Acked 3019 Read 8 Done New Done true 2021 Acked 3020 Write 9 Done New Done true 2022 Acked 3021 Read 5 Done New Done true 2023 Acked 3022 Read 1 Done New Done true 2024 Acked 3023 Write 7 Done New Done true 2025 Acked 3024 Read 5 Done New Done true 2026 Acked 3025 Write 2 Done New Done true 2027 Acked 3026 Write 7 Done New Done true 2028 Acked 3027 Read 7 Done New Done true 2029 Acked 3028 Read 3 Done New Done true 2030 Acked 3029 Write 1 Done New Done true 2031 Acked 3030 Read 2 Done New Done true 2032 Acked 3031 Flush 0 Done New Done true 2033 Acked 3032 Flush 0 Done New Done true 2034 Acked 3033 Read 4 Done New Done true 2035 Acked 3034 Write 10 Done New Done true 2036 Acked 3035 Read 1 Done New Done true 2037 Acked 3036 Flush 0 Done New Done true 2038 Acked 3037 Read 5 Done New Done true 2039 Acked 3038 Read 10 Done New Done true 2040 Acked 3039 Write 8 Done New Done true 2041 Acked 3040 Read 1 Done New Done true 2042 Acked 3041 Write 8 Done New Done true 2043 Acked 3042 Read 10 Done New Done true 2044 Acked 3043 Read 1 Done New Done true 2045 Acked 3044 Read 7 Done New Done true 2046 Acked 3045 Write 4 Done New Done true 2047 Acked 3046 Flush 0 Done New Done true 2048 Acked 3047 Write 1 Done New Done true 2049 Acked 3048 Read 8 Done New Done true 2050 Acked 3049 Write 5 Done New Done true 2051 Acked 3050 Read 7 Done New Done true 2052 Acked 3051 Read 3 Done New Done true 2053 Acked 3052 Flush 0 Done New Done true 2054 Acked 3053 Write 6 Done New Done true 2055 Acked 3054 Read 9 Done New Done true 2056 Acked 3055 Write 6 Done New Done true 2057 Acked 3056 Read 10 Done New Done true 2058 Acked 3057 Write 2 Done New Done true 2059 Acked 3058 Read 6 Done New Done true 2060 Acked 3059 Read 2 Done New Done true 2061 Acked 3060 Flush 0 Done New Done true 2062 Acked 3061 Write 5 Done New Done true 2063 Acked 3062 Read 2 Done New Done true 2064 Acked 3063 Read 10 Done New Done true 2065 Acked 3064 Flush 0 Done New Done true 2066 Acked 3065 Flush 0 Done New Done true 2067 Acked 3066 Read 9 Done New Done true 2068 Acked 3067 Read 5 Done New Done true 2069 Acked 3068 Read 1 Done New Done true 2070 Acked 3069 Write 8 Done New Done true 2071 Acked 3070 Read 6 Done New Done true 2072 Acked 3071 Write 10 Done New Done true 2073 Acked 3072 Write 6 Done New Done true 2074 Acked 3073 Write 3 Done New Done true 2075 Acked 3074 Flush 0 Done New Done true 2076 Acked 3075 Write 2 Done New Done true 2077 Acked 3076 Write 4 Done New Done true 2078 Acked 3077 Read 1 Done New Done true 2079 Acked 3078 Write 8 Done New Done true 2080 Acked 3079 Flush 0 Done New Done true 2081 Acked 3080 Read 10 Done New Done true 2082 Acked 3081 Read 1 Done New Done true 2083 Acked 3082 Read 10 Done New Done true 2084 Acked 3083 Write 6 Done New Done true 2085 Acked 3084 Read 2 Done New Done true 2086 Acked 3085 Write 3 Done New Done true 2087 Acked 3086 Read 10 Done New Done true 2088 Acked 3087 Read 1 Done New Done true 2089 Acked 3088 Read 5 Done New Done true 2090 Acked 3089 Read 8 Done New Done true 2091 Acked 3090 Write 9 Done New Done true 2092 Acked 3091 Read 9 Done New Done true 2093 Acked 3092 Flush 0 Done New Done true 2094 Acked 3093 Read 2 Done New Done true 2095 Acked 3094 Write 8 Done New Done true 2096 Acked 3095 Read 1 Done New Done true 2097 Acked 3096 Read 6 Done New Done true 2098 Acked 3097 Write 4 Done New Done true 2099 Acked 3098 Read 8 Done New Done true 2100 Acked 3099 Write 8 Done New Done true 2101 Acked 3100 Read 5 Done New Done true 2102 Acked 3101 Read 5 Done New Done true 2103 Acked 3102 Read 9 Done New Done true 2104 Acked 3103 Flush 0 Done New Done true 2105 Acked 3104 Write 2 Done New Done false 2106 Acked 3105 Read 1 Done New Done false 2107 Acked 3106 Write 5 Done New Done false 2108 Acked 3107 Write 5 Done New Done false 2109 Acked 3108 Write 9 Done New Done false 2110 Acked 3109 Write 10 Done New Done false 2111 Acked 3110 Write 4 Done New Done false 2112 Acked 3111 Read 4 Done New Done false 2113 Acked 3112 Write 1 Done New Done false 2114 Acked 3113 Write 4 Done New Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 302 0 302 Sent 0 0 0 0 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3103 2811 3103 Downstairs last five completed: 2811 2810 2809 2808 2807 Upstairs last five completed: 2114 2113 2112 2111 2110 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:45.954557457Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:45.954693951Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:45.954723224Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:45.954754898Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:45.954776277Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:45.954800588Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:45.955052943Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:46.955894113Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:46.955960708Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:46.955988658Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:46.956012795Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:46.956038809Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:46.956059003Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:46.956081211Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:46.956317561Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:47.957262939Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:47.957329327Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:47.957357071Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:47.957383006Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:47.957403014Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:47.957439147Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:47.957638171Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:48.95843059Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:48.958495199Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:48.958522899Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:48.958549203Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:48.95856952Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:48.958599688Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:48.958795079Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 1813 Acked 2812 Write 5 Done New Done true 1814 Acked 2813 Flush 0 Done New Done true 1815 Acked 2814 Write 3 Done New Done true 1816 Acked 2815 Read 5 Done New Done true 1817 Acked 2816 Read 9 Done New Done true 1818 Acked 2817 Read 5 Done New Done true 1819 Acked 2818 Read 10 Done New Done true 1820 Acked 2819 Write 8 Done New Done true 1821 Acked 2820 Read 8 Done New Done true 1822 Acked 2821 Write 2 Done New Done true 1823 Acked 2822 Flush 0 Done New Done true 1824 Acked 2823 Read 8 Done New Done true 1825 Acked 2824 Read 2 Done New Done true 1826 Acked 2825 Read 8 Done New Done true 1827 Acked 2826 Write 1 Done New Done true 1828 Acked 2827 Write 10 Done New Done true 1829 Acked 2828 Write 5 Done New Done true 1830 Acked 2829 Read 6 Done New Done true 1831 Acked 2830 Read 5 Done New Done true 1832 Acked 2831 Read 7 Done New Done true 1833 Acked 2832 Write 10 Done New Done true 1834 Acked 2833 Write 5 Done New Done true 1835 Acked 2834 Flush 0 Done New Done true 1836 Acked 2835 Read 6 Done New Done true 1837 Acked 2836 Flush 0 Done New Done true 1838 Acked 2837 Write 9 Done New Done true 1839 Acked 2838 Read 4 Done New Done true 1840 Acked 2839 Read 10 Done New Done true 1841 Acked 2840 Write 7 Done New Done true 1842 Acked 2841 Read 9 Done New Done true 1843 Acked 2842 Read 1 Done New Done true 1844 Acked 2843 Read 4 Done New Done true 1845 Acked 2844 Write 10 Done New Done true 1846 Acked 2845 Write 5 Done New Done true 1847 Acked 2846 Read 9 Done New Done true 1848 Acked 2847 Write 7 Done New Done true 1849 Acked 2848 Write 2 Done New Done true 1850 Acked 2849 Read 10 Done New Done true 1851 Acked 2850 Write 5 Done New Done true 1852 Acked 2851 Read 5 Done New Done true 1853 Acked 2852 Read 2 Done New Done true 1854 Acked 2853 Write 4 Done New Done true 1855 Acked 2854 Write 8 Done New Done true 1856 Acked 2855 Read 4 Done New Done true 1857 Acked 2856 Write 10 Done New Done true 1858 Acked 2857 Write 9 Done New Done true 1859 Acked 2858 Read 9 Done New Done true 1860 Acked 2859 Read 1 Done New Done true 1861 Acked 2860 Flush 0 Done New Done true 1862 Acked 2861 Read 10 Done New Done true 1863 Acked 2862 Read 6 Done New Done true 1864 Acked 2863 Write 1 Done New Done true 1865 Acked 2864 Write 2 Done New Done true 1866 Acked 2865 Read 7 Done New Done true 1867 Acked 2866 Write 7 Done New Done true 1868 Acked 2867 Write 3 Done New Done true 1869 Acked 2868 Read 7 Done New Done true 1870 Acked 2869 Read 6 Done New Done true 1871 Acked 2870 Flush 0 Done New Done true 1872 Acked 2871 Read 5 Done New Done true 1873 Acked 2872 Read 10 Done New Done true 1874 Acked 2873 Read 8 Done New Done true 1875 Acked 2874 Read 7 Done New Done true 1876 Acked 2875 Read 9 Done New Done true 1877 Acked 2876 Flush 0 Done New Done true 1878 Acked 2877 Read 4 Done New Done true 1879 Acked 2878 Read 5 Done New Done true 1880 Acked 2879 Read 7 Done New Done true 1881 Acked 2880 Write 5 Done New Done true 1882 Acked 2881 Write 10 Done New Done true 1883 Acked 2882 Write 9 Done New Done true 1884 Acked 2883 Flush 0 Done New Done true 1885 Acked 2884 Read 3 Done New Done true 1886 Acked 2885 Read 4 Done New Done true 1887 Acked 2886 Read 5 Done New Done true 1888 Acked 2887 Read 6 Done New Done true 1889 Acked 2888 Read 5 Done New Done true 1890 Acked 2889 Read 5 Done New Done true 1891 Acked 2890 Flush 0 Done New Done true 1892 Acked 2891 Read 7 Done New Done true 1893 Acked 2892 Read 9 Done New Done true 1894 Acked 2893 Read 10 Done New Done true 1895 Acked 2894 Write 8 Done New Done true 1896 Acked 2895 Read 6 Done New Done true 1897 Acked 2896 Read 4 Done New Done true 1898 Acked 2897 Write 10 Done New Done true 1899 Acked 2898 Read 1 Done New Done true 1900 Acked 2899 Flush 0 Done New Done true 1901 Acked 2900 Read 3 Done New Done true 1902 Acked 2901 Write 7 Done New Done true 1903 Acked 2902 Write 6 Done New Done true 1904 Acked 2903 Write 3 Done New Done true 1905 Acked 2904 Read 10 Done New Done true 1906 Acked 2905 Read 7 Done New Done true 1907 Acked 2906 Flush 0 Done New Done true 1908 Acked 2907 Write 2 Done New Done true 1909 Acked 2908 Flush 0 Done New Done true 1910 Acked 2909 Write 8 Done New Done true 1911 Acked 2910 Write 5 Done New Done true 1912 Acked 2911 Read 8 Done New Done true 1913 Acked 2912 Read 9 Done New Done true 1914 Acked 2913 Write 4 Done New Done true 1915 Acked 2914 Read 6 Done New Done true 1916 Acked 2915 Write 8 Done New Done true 1917 Acked 2916 Write 10 Done New Done true 1918 Acked 2917 Flush 0 Done New Done true 1919 Acked 2918 Read 4 Done New Done true 1920 Acked 2919 Read 7 Done New Done true 1921 Acked 2920 Write 10 Done New Done true 1922 Acked 2921 Write 7 Done New Done true 1923 Acked 2922 Read 10 Done New Done true 1924 Acked 2923 Read 8 Done New Done true 1925 Acked 2924 Read 3 Done New Done true 1926 Acked 2925 Read 9 Done New Done true 1927 Acked 2926 Write 8 Done New Done true 1928 Acked 2927 Write 1 Done New Done true 1929 Acked 2928 Read 6 Done New Done true 1930 Acked 2929 Read 5 Done New Done true 1931 Acked 2930 Flush 0 Done New Done true 1932 Acked 2931 Write 10 Done New Done true 1933 Acked 2932 Write 2 Done New Done true 1934 Acked 2933 Read 7 Done New Done true 1935 Acked 2934 Read 3 Done New Done true 1936 Acked 2935 Read 5 Done New Done true 1937 Acked 2936 Flush 0 Done New Done true 1938 Acked 2937 Read 4 Done New Done true 1939 Acked 2938 Write 7 Done New Done true 1940 Acked 2939 Read 3 Done New Done true 1941 Acked 2940 Read 3 Done New Done true 1942 Acked 2941 Read 5 Done New Done true 1943 Acked 2942 Read 9 Done New Done true 1944 Acked 2943 Write 8 Done New Done true 1945 Acked 2944 Flush 0 Done New Done true 1946 Acked 2945 Read 6 Done New Done true 1947 Acked 2946 Read 3 Done New Done true 1948 Acked 2947 Write 8 Done New Done true 1949 Acked 2948 Write 6 Done New Done true 1950 Acked 2949 Read 4 Done New Done true 1951 Acked 2950 Flush 0 Done New Done true 1952 Acked 2951 Read 9 Done New Done true 1953 Acked 2952 Read 2 Done New Done true 1954 Acked 2953 Read 6 Done New Done true 1955 Acked 2954 Flush 0 Done New Done true 1956 Acked 2955 Write 10 Done New Done true 1957 Acked 2956 Read 10 Done New Done true 1958 Acked 2957 Write 9 Done New Done true 1959 Acked 2958 Read 4 Done New Done true 1960 Acked 2959 Read 9 Done New Done true 1961 Acked 2960 Read 1 Done New Done true 1962 Acked 2961 Read 3 Done New Done true 1963 Acked 2962 Read 2 Done New Done true 1964 Acked 2963 Write 10 Done New Done true 1965 Acked 2964 Write 3 Done New Done true 1966 Acked 2965 Read 2 Done New Done true 1967 Acked 2966 Read 9 Done New Done true 1968 Acked 2967 Read 9 Done New Done true 1969 Acked 2968 Read 10 Done New Done true 1970 Acked 2969 Read 1 Done New Done true 1971 Acked 2970 Read 8 Done New Done true 1972 Acked 2971 Read 8 Done New Done true 1973 Acked 2972 Write 6 Done New Done true 1974 Acked 2973 Read 6 Done New Done true 1975 Acked 2974 Write 10 Done New Done true 1976 Acked 2975 Write 7 Done New Done true 1977 Acked 2976 Write 5 Done New Done true 1978 Acked 2977 Flush 0 Done New Done true 1979 Acked 2978 Flush 0 Done New Done true 1980 Acked 2979 Write 6 Done New Done true 1981 Acked 2980 Write 9 Done New Done true 1982 Acked 2981 Write 10 Done New Done true 1983 Acked 2982 Write 5 Done New Done true 1984 Acked 2983 Read 3 Done New Done true 1985 Acked 2984 Read 3 Done New Done true 1986 Acked 2985 Flush 0 Done New Done true 1987 Acked 2986 Read 9 Done New Done true 1988 Acked 2987 Read 5 Done New Done true 1989 Acked 2988 Write 6 Done New Done true 1990 Acked 2989 Write 9 Done New Done true 1991 Acked 2990 Read 5 Done New Done true 1992 Acked 2991 Read 4 Done New Done true 1993 Acked 2992 Write 8 Done New Done true 1994 Acked 2993 Flush 0 Done New Done true 1995 Acked 2994 Write 4 Done New Done true 1996 Acked 2995 Read 9 Done New Done true 1997 Acked 2996 Write 9 Done New Done true 1998 Acked 2997 Read 1 Done New Done true 1999 Acked 2998 Read 6 Done New Done true 2000 Acked 2999 Write 6 Done New Done true 2001 Acked 3000 Read 9 Done New Done true 2002 Acked 3001 Read 8 Done New Done true 2003 Acked 3002 Read 3 Done New Done true 2004 Acked 3003 Write 2 Done New Done true 2005 Acked 3004 Write 7 Done New Done true 2006 Acked 3005 Read 10 Done New Done true 2007 Acked 3006 Read 6 Done New Done true 2008 Acked 3007 Write 9 Done New Done true 2009 Acked 3008 Read 4 Done New Done true 2010 Acked 3009 Write 10 Done New Done true 2011 Acked 3010 Read 8 Done New Done true 2012 Acked 3011 Read 10 Done New Done true 2013 Acked 3012 Read 3 Done New Done true 2014 Acked 3013 Read 9 Done New Done true 2015 Acked 3014 Write 6 Done New Done true 2016 Acked 3015 Read 1 Done New Done true 2017 Acked 3016 Write 9 Done New Done true 2018 Acked 3017 Write 10 Done New Done true 2019 Acked 3018 Write 10 Done New Done true 2020 Acked 3019 Read 8 Done New Done true 2021 Acked 3020 Write 9 Done New Done true 2022 Acked 3021 Read 5 Done New Done true 2023 Acked 3022 Read 1 Done New Done true 2024 Acked 3023 Write 7 Done New Done true 2025 Acked 3024 Read 5 Done New Done true 2026 Acked 3025 Write 2 Done New Done true 2027 Acked 3026 Write 7 Done New Done true 2028 Acked 3027 Read 7 Done New Done true 2029 Acked 3028 Read 3 Done New Done true 2030 Acked 3029 Write 1 Done New Done true 2031 Acked 3030 Read 2 Done New Done true 2032 Acked 3031 Flush 0 Done New Done true 2033 Acked 3032 Flush 0 Done New Done true 2034 Acked 3033 Read 4 Done New Done true 2035 Acked 3034 Write 10 Done New Done true 2036 Acked 3035 Read 1 Done New Done true 2037 Acked 3036 Flush 0 Done New Done true 2038 Acked 3037 Read 5 Done New Done true 2039 Acked 3038 Read 10 Done New Done true 2040 Acked 3039 Write 8 Done New Done true 2041 Acked 3040 Read 1 Done New Done true 2042 Acked 3041 Write 8 Done New Done true 2043 Acked 3042 Read 10 Done New Done true 2044 Acked 3043 Read 1 Done New Done true 2045 Acked 3044 Read 7 Done New Done true 2046 Acked 3045 Write 4 Done New Done true 2047 Acked 3046 Flush 0 Done New Done true 2048 Acked 3047 Write 1 Done New Done true 2049 Acked 3048 Read 8 Done New Done true 2050 Acked 3049 Write 5 Done New Done true 2051 Acked 3050 Read 7 Done New Done true 2052 Acked 3051 Read 3 Done New Done true 2053 Acked 3052 Flush 0 Done New Done true 2054 Acked 3053 Write 6 Done New Done true 2055 Acked 3054 Read 9 Done New Done true 2056 Acked 3055 Write 6 Done New Done true 2057 Acked 3056 Read 10 Done New Done true 2058 Acked 3057 Write 2 Done New Done true 2059 Acked 3058 Read 6 Done New Done true 2060 Acked 3059 Read 2 Done New Done true 2061 Acked 3060 Flush 0 Done New Done true 2062 Acked 3061 Write 5 Done New Done true 2063 Acked 3062 Read 2 Done New Done true 2064 Acked 3063 Read 10 Done New Done true 2065 Acked 3064 Flush 0 Done New Done true 2066 Acked 3065 Flush 0 Done New Done true 2067 Acked 3066 Read 9 Done New Done true 2068 Acked 3067 Read 5 Done New Done true 2069 Acked 3068 Read 1 Done New Done true 2070 Acked 3069 Write 8 Done New Done true 2071 Acked 3070 Read 6 Done New Done true 2072 Acked 3071 Write 10 Done New Done true 2073 Acked 3072 Write 6 Done New Done true 2074 Acked 3073 Write 3 Done New Done true 2075 Acked 3074 Flush 0 Done New Done true 2076 Acked 3075 Write 2 Done New Done true 2077 Acked 3076 Write 4 Done New Done true 2078 Acked 3077 Read 1 Done New Done true 2079 Acked 3078 Write 8 Done New Done true 2080 Acked 3079 Flush 0 Done New Done true 2081 Acked 3080 Read 10 Done New Done true 2082 Acked 3081 Read 1 Done New Done true 2083 Acked 3082 Read 10 Done New Done true 2084 Acked 3083 Write 6 Done New Done true 2085 Acked 3084 Read 2 Done New Done true 2086 Acked 3085 Write 3 Done New Done true 2087 Acked 3086 Read 10 Done New Done true 2088 Acked 3087 Read 1 Done New Done true 2089 Acked 3088 Read 5 Done New Done true 2090 Acked 3089 Read 8 Done New Done true 2091 Acked 3090 Write 9 Done New Done true 2092 Acked 3091 Read 9 Done New Done true 2093 Acked 3092 Flush 0 Done New Done true 2094 Acked 3093 Read 2 Done New Done true 2095 Acked 3094 Write 8 Done New Done true 2096 Acked 3095 Read 1 Done New Done true 2097 Acked 3096 Read 6 Done New Done true 2098 Acked 3097 Write 4 Done New Done true 2099 Acked 3098 Read 8 Done New Done true 2100 Acked 3099 Write 8 Done New Done true 2101 Acked 3100 Read 5 Done New Done true 2102 Acked 3101 Read 5 Done New Done true 2103 Acked 3102 Read 9 Done New Done true 2104 Acked 3103 Flush 0 Done New Done true 2105 Acked 3104 Write 2 Done New Done true 2106 Acked 3105 Read 1 Done New Done true 2107 Acked 3106 Write 5 Done New Done true 2108 Acked 3107 Write 5 Done New Done true 2109 Acked 3108 Write 9 Done New Done true 2110 Acked 3109 Write 10 Done New Done true 2111 Acked 3110 Write 4 Done New Done true 2112 Acked 3111 Read 4 Done New Done true 2113 Acked 3112 Write 1 Done New Done true 2114 Acked 3113 Write 4 Done New Done true 2115 Acked 3114 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 303 0 303 Sent 0 0 0 0 Done 303 0 303 606 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3114 2811 3114 Downstairs last five completed: 2811 2810 2809 2808 2807 Upstairs last five completed: 2115 2114 2113 2112 2111 CLIENT: Up:0 ds:303 act:2 [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:49.959900125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:49.959996638Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:49.960046592Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:49.960095877Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:49.960133459Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 303 jobs since flush 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:49.960173747Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:49.960429732Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1192) [8820][1] reports Running {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.961441718Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.961537889Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.96354135Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.963577683Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.963601496Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 2811","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.963886909Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.963921154Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.963944279Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.963966448Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:50.963988722Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 3114 3113 3112 3111 3110 Upstairs last five completed: 2115 2114 2113 2112 2111 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(2) stop 2 Jan 02 22:31:52.973 INFO request completed, latency_us: 448, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 49f1360d-075b-46a5-a7c0-e9d096462565, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 [8830] Got stop action so:false kr:false Jan 02 22:31:52.974 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/2, method: GET, req_id: 588e9811-09b6-4d7b-9cac-686c349bcbf0, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:52.975125733Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:52.975305563Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:52.975348956Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:52.975371988Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:52.975407726Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:52.975451067Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:53.976756072Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:53.976826093Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:53.976854385Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:53.976880895Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:53.976901372Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:53.9769401Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:53.976988061Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:54.97906908Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:54.979166698Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:54.979295021Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:54.979429026Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:54.979456984Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:54.979494073Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:54.979644484Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:55.98064784Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:55.980716125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:55.980751612Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:55.980790426Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:55.980812779Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:55.980835565Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:55.98087947Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:31:56.977 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/2, method: GET, req_id: b69dd9fe-201a-442e-8063-9f30db55f3a8, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 2706 len 12288 002/300 Read block 837 len 40960 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:56.981078655Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:56.981152299Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:56.981198188Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:56.981234755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:56.981284158Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:56.981323286Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 2 jobs since flush 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:56.981357884Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:56.981381255Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Write block 922 len 28672 data: 2 2 2 2 2 2 2 004/300 Flush 005/300 Read block 574 len 24576 006/300 Flush 007/300 Write block 2838 len 12288 data: 3 3 3 008/300 Write block 1362 len 8192 data: 3 3 009/300 Read block 1051 len 24576 010/300 Write block 3384 len 32768 data: 3 2 2 2 2 2 2 2 011/300 Read block 3832 len 16384 012/300 Read block 3724 len 32768 013/300 Flush 014/300 Read block 2172 len 12288 015/300 Read block 2831 len 12288 016/300 Write block 4952 len 28672 data: 2 2 2 2 2 3 3 017/300 Write block 1077 len 36864 data: 3 2 2 2 2 2 2 3 3 018/300 Write block 4730 len 4096 data: 5 019/300 Read block 4525 len 28672 020/300 Read block 4075 len 32768 021/300 Write block 1591 len 36864 data: 4 4 3 3 3 3 3 2 2 022/300 Read block 2526 len 12288 023/300 Write block 3541 len 16384 data: 4 4 4 3 024/300 Write block 2563 len 8192 data: 2 2 025/300 Read block 4048 len 36864 026/300 Read block 1323 len 24576 027/300 Read block 1640 len 4096 028/300 Read block 3800 len 40960 029/300 Write block 2724 len 40960 data: 2 2 2 2 3 4 3 3 3 4 030/300 Write block 3827 len 20480 data: 2 2 2 2 2 031/300 Write block 117 len 40960 data: 2 2 2 2 2 2 2 2 3 3 032/300 Write block 3587 len 20480 data: 2 3 3 4 5 033/300 Read block 4614 len 24576 034/300 Read block 1421 len 36864 035/300 Write block 2196 len 24576 data: 3 2 2 2 2 2 036/300 Write block 1371 len 36864 data: 2 2 2 2 2 2 2 2 3 037/300 Read block 2288 len 28672 038/300 Read block 3117 len 28672 039/300 Read block 4501 len 20480 040/300 Write block 727 len 32768 data: 3 3 3 3 4 4 3 3 041/300 Read block 507 len 32768 042/300 Read block 1415 len 36864 043/300 Read block 3035 len 4096 044/300 Write block 2709 len 8192 data: 2 3 045/300 Read block 1990 len 36864 046/300 Write block 4914 len 32768 data: 4 3 2 2 3 3 2 2 047/300 Write block 3056 len 20480 data: 5 5 4 4 3 048/300 Read block 1368 len 32768 049/300 Read block 1800 len 20480 050/300 Read block 3531 len 28672 051/300 Read block 1036 len 8192 052/300 Flush 053/300 Write block 3778 len 12288 data: 2 2 2 054/300 Read block 2637 len 4096 055/300 Write block 1240 len 28672 data: 2 2 2 3 3 3 3 056/300 Write block 2282 len 32768 data: 3 3 3 2 2 2 2 2 057/300 Read block 3176 len 4096 058/300 Read block 4011 len 36864 059/300 Read block 1809 len 28672 060/300 Read block 1574 len 4096 061/300 Write block 1159 len 32768 data: 3 3 3 3 3 3 3 4 062/300 Write block 1556 len 12288 data: 2 2 3 063/300 Flush 064/300 Read block 3963 len 32768 065/300 Write block 2007 len 12288 data: 5 7 7 066/300 Write block 4874 len 40960 data: 4 4 4 4 3 4 4 5 5 5 067/300 Write block 4432 len 28672 data: 4 4 4 2 2 3 3 068/300 Read block 1039 len 24576 069/300 Read block 2556 len 20480 070/300 Read block 1647 len 8192 071/300 Flush 072/300 Write block 3424 len 24576 data: 3 3 3 3 3 2 073/300 Write block 4802 len 16384 data: 3 3 3 4 074/300 Write block 932 len 32768 data: 2 2 3 3 3 4 5 5 075/300 Read block 1297 len 16384 076/300 Read block 793 len 40960 077/300 Write block 4090 len 12288 data: 3 3 2 078/300 Read block 921 len 40960 079/300 Write block 4214 len 28672 data: 2 3 3 3 3 3 3 080/300 Read block 3269 len 20480 081/300 Read block 2662 len 8192 082/300 Read block 1830 len 4096 083/300 Read block 1303 len 20480 084/300 Write block 3354 len 28672 data: 2 2 2 2 3 5 5 085/300 Write block 4418 len 36864 data: 2 2 2 2 3 4 4 3 4 086/300 Flush 087/300 Read block 3160 len 32768 088/300 Write block 438 len 20480 data: 5 5 5 4 4 089/300 Write block 1228 len 8192 data: 4 4 090/300 Write block 2643 len 20480 data: 3 4 4 4 2 091/300 Write block 4127 len 24576 data: 2 2 2 2 3 3 092/300 Write block 4259 len 32768 data: 2 2 2 2 2 2 2 4 093/300 Read block 4736 len 40960 094/300 Write block 4090 len 28672 data: 4 4 3 2 3 3 4 095/300 Read block 1952 len 40960 096/300 Read block 2679 len 20480 097/300 Read block 4986 len 32768 098/300 Read block 209 len 20480 099/300 Read block 3469 len 36864 100/300 Read block 4387 len 36864 101/300 Write block 1293 len 24576 data: 3 3 3 2 2 3 102/300 Write block 3695 len 32768 data: 2 2 2 2 2 2 2 2 103/300 Write block 1880 len 16384 data: 2 2 2 2 104/300 Flush 105/300 Read block 3656 len 40960 106/300 Write block 3886 len 40960 data: 5 5 4 2 2 3 3 3 3 4 107/300 Read block 320 len 16384 108/300 Flush 109/300 Read block 4595 len 16384 110/300 Read block 1086 len 20480 111/300 Write block 4933 len 12288 data: 4 3 3 112/300 Read block 3212 len 8192 113/300 Write block 2820 len 28672 data: 3 5 5 5 5 6 6 114/300 Read block 3942 len 20480 115/300 Read block 2850 len 20480 116/300 Write block 2770 len 16384 data: 8 7 7 6 117/300 Read block 1351 len 24576 118/300 Read block 1920 len 36864 119/300 Write block 1431 len 32768 data: 2 2 2 2 2 2 2 2 120/300 Read block 305 len 32768 121/300 Write block 4861 len 4096 data: 2 122/300 Read block 2737 len 40960 123/300 Write block 2334 len 20480 data: 4 3 3 4 3 124/300 Write block 3766 len 24576 data: 7 5 5 4 4 4 125/300 Read block 2122 len 16384 126/300 Write block 3265 len 4096 data: 5 127/300 Flush 128/300 Read block 3657 len 8192 129/300 Read block 267 len 24576 130/300 Read block 3222 len 4096 131/300 Write block 3855 len 12288 data: 2 2 3 132/300 Write block 437 len 40960 data: 5 6 6 6 5 5 4 3 4 2 133/300 Read block 2649 len 4096 134/300 Read block 4134 len 4096 135/300 Flush 136/300 Write block 1150 len 4096 data: 2 137/300 Read block 2025 len 24576 138/300 Read block 442 len 40960 139/300 Read block 582 len 28672 140/300 Read block 4855 len 20480 141/300 Write block 94 len 4096 data: 3 142/300 Write block 4004 len 32768 data: 3 3 2 2 2 2 2 3 143/300 Flush 144/300 Flush 145/300 Read block 3103 len 4096 146/300 Read block 472 len 8192 147/300 Read block 2808 len 32768 148/300 Write block 3496 len 20480 data: 2 3 2 3 3 149/300 Read block 669 len 32768 150/300 Read block 4596 len 24576 151/300 Read block 983 len 8192 152/300 Write block 4561 len 40960 data: 3 3 3 3 2 2 2 2 2 2 153/300 Write block 3995 len 36864 data: 6 5 5 4 2 2 2 2 4 154/300 Read block 4446 len 28672 155/300 Read block 1527 len 20480 156/300 Read block 4935 len 16384 157/300 Read block 3141 len 40960 158/300 Read block 4773 len 4096 159/300 Flush 160/300 Write block 3525 len 32768 data: 2 2 3 2 2 2 2 2 161/300 Read block 2398 len 36864 162/300 Write block 2300 len 20480 data: 3 3 3 3 3 163/300 Read block 4802 len 36864 164/300 Read block 589 len 16384 165/300 Write block 4770 len 24576 data: 4 5 5 4 3 3 166/300 Read block 987 len 40960 167/300 Write block 1353 len 8192 data: 5 4 168/300 Read block 902 len 24576 169/300 Read block 4034 len 4096 170/300 Read block 3406 len 28672 171/300 Read block 1790 len 12288 172/300 Read block 3208 len 8192 173/300 Flush 174/300 Write block 2773 len 12288 data: 7 4 4 175/300 Write block 1389 len 16384 data: 4 4 4 6 176/300 Write block 3885 len 24576 data: 5 6 6 5 3 3 177/300 Write block 459 len 28672 data: 3 3 3 3 3 2 2 178/300 Write block 660 len 24576 data: 3 3 3 5 5 5 179/300 Write block 4335 len 40960 data: 3 3 3 2 3 3 4 4 5 5 180/300 Read block 2848 len 16384 181/300 Write block 3714 len 4096 data: 2 182/300 Read block 4554 len 24576 183/300 Write block 10 len 28672 data: 3 4 3 3 4 5 5 184/300 Flush 185/300 Write block 4699 len 24576 data: 3 3 2 2 2 2 186/300 Flush 187/300 Read block 1100 len 28672 188/300 Write block 1141 len 8192 data: 3 2 189/300 Read block 1597 len 8192 190/300 Read block 525 len 32768 191/300 Write block 4470 len 8192 data: 2 2 192/300 Flush 193/300 Read block 3285 len 4096 194/300 Read block 2783 len 16384 195/300 Flush 196/300 Read block 131 len 36864 197/300 Write block 3333 len 20480 data: 4 3 3 3 3 198/300 Write block 3066 len 8192 data: 3 3 199/300 Read block 2828 len 40960 200/300 Read block 4832 len 36864 201/300 Read block 815 len 28672 202/300 Write block 336 len 12288 data: 2 2 3 203/300 Write block 2433 len 36864 data: 2 2 2 2 2 3 3 2 2 204/300 Read block 3874 len 16384 205/300 Read block 1156 len 4096 206/300 Read block 1528 len 4096 207/300 Write block 2780 len 40960 data: 4 3 3 3 3 3 2 2 2 2 208/300 Read block 3387 len 8192 209/300 Read block 3825 len 32768 210/300 Read block 2071 len 28672 211/300 Write block 1039 len 12288 data: 2 4 2 212/300 Read block 2932 len 4096 213/300 Flush 214/300 Read block 4595 len 4096 215/300 Write block 808 len 8192 data: 2 2 216/300 Flush 217/300 Read block 2794 len 20480 218/300 Read block 2589 len 36864 219/300 Read block 3488 len 8192 220/300 Write block 3416 len 40960 data: 2 3 3 3 3 3 3 3 4 4 221/300 Read block 1915 len 28672 222/300 Write block 4446 len 12288 data: 4 4 4 223/300 Write block 4616 len 8192 data: 3 2 224/300 Read block 3336 len 36864 225/300 Read block 1192 len 4096 226/300 Write block 3236 len 8192 data: 3 4 227/300 Write block 2121 len 16384 data: 3 3 3 3 228/300 Write block 1298 len 32768 data: 4 3 3 2 2 2 2 2 229/300 Write block 3228 len 40960 data: 3 3 3 3 3 3 2 3 4 5 230/300 Write block 1533 len 16384 data: 2 2 2 2 231/300 Flush 232/300 Read block 623 len 8192 233/300 Read block 4217 len 4096 234/300 Read block 4477 len 40960 235/300 Write block 3689 len 40960 data: 2 2 3 3 3 3 3 3 3 3 236/300 Read block 4419 len 12288 237/300 Write block 4877 len 24576 data: 5 4 5 5 6 6 238/300 Flush 239/300 Write block 205 len 20480 data: 2 2 2 2 2 240/300 Read block 4044 len 32768 241/300 Read block 4153 len 12288 242/300 Read block 4278 len 24576 243/300 Read block 4990 len 4096 244/300 Write block 4574 len 40960 data: 2 2 2 2 2 3 3 3 3 3 245/300 Read block 536 len 12288 246/300 Write block 3377 len 8192 data: 3 3 247/300 Write block 1050 len 20480 data: 2 2 2 2 2 248/300 Write block 238 len 32768 data: 3 3 2 2 2 2 2 3 249/300 Write block 3197 len 4096 data: 2 250/300 Write block 4364 len 8192 data: 2 2 251/300 Write block 3875 len 36864 data: 2 2 2 2 2 2 4 5 5 252/300 Flush 253/300 Write block 3259 len 12288 data: 5 5 4 254/300 Flush 255/300 Read block 3569 len 36864 256/300 Read block 743 len 12288 257/300 Read block 4447 len 28672 258/300 Read block 2482 len 40960 259/300 Flush 260/300 Read block 1910 len 20480 261/300 Flush 262/300 Flush 263/300 Write block 4538 len 36864 data: 4 4 4 4 4 3 3 2 2 264/300 Read block 3852 len 36864 265/300 Read block 3567 len 36864 266/300 Write block 161 len 20480 data: 4 3 2 2 2 267/300 Read block 1885 len 24576 268/300 Write block 130 len 40960 data: 4 4 3 3 4 4 5 4 4 2 269/300 Flush 270/300 Read block 3842 len 4096 271/300 Read block 3650 len 24576 272/300 Read block 842 len 32768 273/300 Read block 1796 len 24576 274/300 Read block 3862 len 40960 275/300 Write block 2196 len 24576 data: 4 3 3 3 3 3 276/300 Read block 3452 len 36864 277/300 Flush 278/300 Write block 3602 len 40960 data: 4 4 3 3 3 3 3 4 4 4 279/300 Read block 2065 len 24576 280/300 Read block 3835 len 4096 281/300 Read block 4617 len 36864 282/300 Read block 3055 len 20480 283/300 Read block 1514 len 16384 284/300 Read block 4554 len 36864 285/300 Write block 1327 len 36864 data: 4 3 3 3 3 2 3 3 3 286/300 Flush 287/300 Read block 3336 len 36864 288/300 Read block 2288 len 40960 289/300 Read block 2605 len 16384 290/300 Write block 105 len 8192 data: 3 3 291/300 Read block 1595 len 16384 292/300 Flush 293/300 Read block 345 len 4096 294/300 Read block 1483 len 28672 295/300 Write block 4823 len 8192 data: 3 3 296/300 Read block 4017 len 24576 297/300 Write block 94 len 32768 data: 4 3 3 3 3 3 3 3 298/300 Write block 3525 len 40960 data: 3 3 4 3 3 3 3 3 2 2 299/300 Flush 300/300 Read block 2361 len 4096 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:31:57.965 INFO request completed, latency_us: 287, response_code: 204, uri: /start/cid/2, method: POST, req_id: c9f7302a-39ec-4125-beba-c41f0e495463, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2116 Acked 3115 Read 3 Done Done New true 2117 Acked 3116 Read 10 Done Done New true 2118 Acked 3117 Write 7 Done Done New false 2119 Acked 3118 Flush 0 Done Done New false 2120 Acked 3119 Read 6 Done Done New false 2121 Acked 3120 Flush 0 Done Done New false 2122 Acked 3121 Write 3 Done Done New false 2123 Acked 3122 Write 2 Done Done New false 2124 Acked 3123 Read 6 Done Done New false 2125 Acked 3124 Write 8 Done Done New false 2126 Acked 3125 Read 4 Done Done New false 2127 Acked 3126 Read 8 Done Done New false 2128 Acked 3127 Flush 0 Done Done New false 2129 Acked 3128 Read 3 Done Done New false 2130 Acked 3129 Read 3 Done Done New false 2131 Acked 3130 Write 7 Done Done New false 2132 Acked 3131 Write 9 Done Done New false 2133 Acked 3132 Write 1 Done Done New false 2134 Acked 3133 Read 7 Done Done New false 2135 Acked 3134 Read 8 Done Done New false 2136 Acked 3135 Write 9 Done Done New false 2137 Acked 3136 Read 3 Done Done New false 2138 Acked 3137 Write 4 Done Done New false 2139 Acked 3138 Write 2 Done Done New false 2140 Acked 3139 Read 9 Done Done New false 2141 Acked 3140 Read 6 Done Done New false 2142 Acked 3141 Read 1 Done Done New false 2143 Acked 3142 Read 10 Done Done New false 2144 Acked 3143 Write 10 Done Done New false 2145 Acked 3144 Write 5 Done Done New false 2146 Acked 3145 Write 10 Done Done New false 2147 Acked 3146 Write 5 Done Done New false 2148 Acked 3147 Read 6 Done Done New false 2149 Acked 3148 Read 9 Done Done New false 2150 Acked 3149 Write 6 Done Done New false 2151 Acked 3150 Write 9 Done Done New false 2152 Acked 3151 Read 7 Done Done New false 2153 Acked 3152 Read 7 Done Done New false 2154 Acked 3153 Read 5 Done Done New false 2155 Acked 3154 Write 8 Done Done New false 2156 Acked 3155 Read 8 Done Done New false 2157 Acked 3156 Read 9 Done Done New false 2158 Acked 3157 Read 1 Done Done New false 2159 Acked 3158 Write 2 Done Done New false 2160 Acked 3159 Read 9 Done Done New false 2161 Acked 3160 Write 8 Done Done New false 2162 Acked 3161 Write 5 Done Done New false 2163 Acked 3162 Read 8 Done Done New false 2164 Acked 3163 Read 5 Done Done New false 2165 Acked 3164 Read 7 Done Done New false 2166 Acked 3165 Read 2 Done Done New false 2167 Acked 3166 Flush 0 Done Done New false 2168 Acked 3167 Write 3 Done Done New false 2169 Acked 3168 Read 1 Done Done New false 2170 Acked 3169 Write 7 Done Done New false 2171 Acked 3170 Write 8 Done Done New false 2172 Acked 3171 Read 1 Done Done New false 2173 Acked 3172 Read 9 Done Done New false 2174 Acked 3173 Read 7 Done Done New false 2175 Acked 3174 Read 1 Done Done New false 2176 Acked 3175 Write 8 Done Done New false 2177 Acked 3176 Write 3 Done Done New false 2178 Acked 3177 Flush 0 Done Done New false 2179 Acked 3178 Read 8 Done Done New false 2180 Acked 3179 Write 3 Done Done New false 2181 Acked 3180 Write 10 Done Done New false 2182 Acked 3181 Write 7 Done Done New false 2183 Acked 3182 Read 6 Done Done New false 2184 Acked 3183 Read 5 Done Done New false 2185 Acked 3184 Read 2 Done Done New false 2186 Acked 3185 Flush 0 Done Done New false 2187 Acked 3186 Write 6 Done Done New false 2188 Acked 3187 Write 4 Done Done New false 2189 Acked 3188 Write 8 Done Done New false 2190 Acked 3189 Read 4 Done Done New false 2191 Acked 3190 Read 10 Done Done New false 2192 Acked 3191 Write 3 Done Done New false 2193 Acked 3192 Read 10 Done Done New false 2194 Acked 3193 Write 7 Done Done New false 2195 Acked 3194 Read 5 Done Done New false 2196 Acked 3195 Read 2 Done Done New false 2197 Acked 3196 Read 1 Done Done New false 2198 Acked 3197 Read 5 Done Done New false 2199 Acked 3198 Write 7 Done Done New false 2200 Acked 3199 Write 9 Done Done New false 2201 Acked 3200 Flush 0 Done Done New false 2202 Acked 3201 Read 8 Done Done New false 2203 Acked 3202 Write 5 Done Done New false 2204 Acked 3203 Write 2 Done Done New false 2205 Acked 3204 Write 5 Done Done New false 2206 Acked 3205 Write 6 Done Done New false 2207 Acked 3206 Write 8 Done Done New false 2208 Acked 3207 Read 10 Done Done New false 2209 Acked 3208 Write 7 Done Done New false 2210 Acked 3209 Read 10 Done Done New false 2211 Acked 3210 Read 5 Done Done New false 2212 Acked 3211 Read 8 Done Done New false 2213 Acked 3212 Read 5 Done Done New false 2214 Acked 3213 Read 9 Done Done New false 2215 Acked 3214 Read 9 Done Done New false 2216 Acked 3215 Write 6 Done Done New false 2217 Acked 3216 Write 8 Done Done New false 2218 Acked 3217 Write 4 Done Done New false 2219 Acked 3218 Flush 0 Done Done New false 2220 Acked 3219 Read 10 Done Done New false 2221 Acked 3220 Write 10 Done Done New false 2222 Acked 3221 Read 4 Done Done New false 2223 Acked 3222 Flush 0 Done Done New false 2224 Acked 3223 Read 4 Done Done New false 2225 Acked 3224 Read 5 Done Done New false 2226 Acked 3225 Write 3 Done Done New false 2227 Acked 3226 Read 2 Done Done New false 2228 Acked 3227 Write 7 Done Done New false 2229 Acked 3228 Read 5 Done Done New false 2230 Acked 3229 Read 5 Done Done New false 2231 Acked 3230 Write 4 Done Done New false 2232 Acked 3231 Read 6 Done Done New false 2233 Acked 3232 Read 9 Done Done New false 2234 Acked 3233 Write 8 Done Done New false 2235 Acked 3234 Read 8 Done Done New false 2236 Acked 3235 Write 1 Done Done New false 2237 Acked 3236 Read 10 Done Done New false 2238 Acked 3237 Write 5 Done Done New false 2239 Acked 3238 Write 6 Done Done New false 2240 Acked 3239 Read 4 Done Done New false 2241 Acked 3240 Write 1 Done Done New false 2242 Acked 3241 Flush 0 Done Done New false 2243 Acked 3242 Read 2 Done Done New false 2244 Acked 3243 Read 6 Done Done New false 2245 Acked 3244 Read 1 Done Done New false 2246 Acked 3245 Write 3 Done Done New false 2247 Acked 3246 Write 10 Done Done New false 2248 Acked 3247 Read 1 Done Done New false 2249 Acked 3248 Read 1 Done Done New false 2250 Acked 3249 Flush 0 Done Done New false 2251 Acked 3250 Write 1 Done Done New false 2252 Acked 3251 Read 6 Done Done New false 2253 Acked 3252 Read 10 Done Done New false 2254 Acked 3253 Flush 0 Done Done New false 2255 Acked 3254 Read 7 Done Done New false 2256 Acked 3255 Read 5 Done Done New false 2257 Acked 3256 Write 1 Done Done New false 2258 Acked 3257 Write 8 Done Done New false 2259 Acked 3258 Flush 0 Done Done New false 2260 Acked 3259 Flush 0 Done Done New false 2261 Acked 3260 Read 1 Done Done New false 2262 Acked 3261 Read 2 Done Done New false 2263 Acked 3262 Read 8 Done Done New false 2264 Acked 3263 Write 5 Done Done New false 2265 Acked 3264 Read 8 Done Done New false 2266 Acked 3265 Read 6 Done Done New false 2267 Acked 3266 Read 2 Done Done New false 2268 Acked 3267 Write 10 Done Done New false 2269 Acked 3268 Write 9 Done Done New false 2270 Acked 3269 Read 7 Done Done New false 2271 Acked 3270 Read 5 Done Done New false 2272 Acked 3271 Read 4 Done Done New false 2273 Acked 3272 Read 10 Done Done New false 2274 Acked 3273 Read 1 Done Done New false 2275 Acked 3274 Flush 0 Done Done New false 2276 Acked 3275 Write 8 Done Done New false 2277 Acked 3276 Read 9 Done Done New false 2278 Acked 3277 Write 5 Done Done New false 2279 Acked 3278 Read 9 Done Done New false 2280 Acked 3279 Read 4 Done Done New false 2281 Acked 3280 Write 6 Done Done New false 2282 Acked 3281 Read 10 Done Done New false 2283 Acked 3282 Write 2 Done Done New false 2284 Acked 3283 Read 6 Done Done New false 2285 Acked 3284 Read 1 Done Done New false 2286 Acked 3285 Read 7 Done Done New false 2287 Acked 3286 Read 3 Done Done New false 2288 Acked 3287 Read 2 Done Done New false 2289 Acked 3288 Flush 0 Done Done New false 2290 Acked 3289 Write 3 Done Done New false 2291 Acked 3290 Write 4 Done Done New false 2292 Acked 3291 Write 6 Done Done New false 2293 Acked 3292 Write 7 Done Done New false 2294 Acked 3293 Write 6 Done Done New false 2295 Acked 3294 Write 10 Done Done New false 2296 Acked 3295 Read 4 Done Done New false 2297 Acked 3296 Write 1 Done Done New false 2298 Acked 3297 Read 6 Done Done New false 2299 Acked 3298 Write 7 Done Done New false 2300 Acked 3299 Flush 0 Done Done New false 2301 Acked 3300 Write 6 Done Done New false 2302 Acked 3301 Flush 0 Done Done New false 2303 Acked 3302 Read 7 Done Done New false 2304 Acked 3303 Write 2 Done Done New false 2305 Acked 3304 Read 2 Done Done New false 2306 Acked 3305 Read 8 Done Done New false 2307 Acked 3306 Write 2 Done Done New false 2308 Acked 3307 Flush 0 Done Done New false 2309 Acked 3308 Read 1 Done Done New false 2310 Acked 3309 Read 4 Done Done New false 2311 Acked 3310 Flush 0 Done Done New false 2312 Acked 3311 Read 9 Done Done New false 2313 Acked 3312 Write 5 Done Done New false 2314 Acked 3313 Write 2 Done Done New false 2315 Acked 3314 Read 10 Done Done New false 2316 Acked 3315 Read 9 Done Done New false 2317 Acked 3316 Read 7 Done Done New false 2318 Acked 3317 Write 3 Done Done New false 2319 Acked 3318 Write 9 Done Done New false 2320 Acked 3319 Read 4 Done Done New false 2321 Acked 3320 Read 1 Done Done New false 2322 Acked 3321 Read 1 Done Done New false 2323 Acked 3322 Write 10 Done Done New false 2324 Acked 3323 Read 2 Done Done New false 2325 Acked 3324 Read 8 Done Done New false 2326 Acked 3325 Read 7 Done Done New false 2327 Acked 3326 Write 3 Done Done New false 2328 Acked 3327 Read 1 Done Done New false 2329 Acked 3328 Flush 0 Done Done New false 2330 Acked 3329 Read 1 Done Done New false 2331 Acked 3330 Write 2 Done Done New false 2332 Acked 3331 Flush 0 Done Done New false 2333 Acked 3332 Read 5 Done Done New false 2334 Acked 3333 Read 9 Done Done New false 2335 Acked 3334 Read 2 Done Done New false 2336 Acked 3335 Write 10 Done Done New false 2337 Acked 3336 Read 7 Done Done New false 2338 Acked 3337 Write 3 Done Done New false 2339 Acked 3338 Write 2 Done Done New false 2340 Acked 3339 Read 9 Done Done New false 2341 Acked 3340 Read 1 Done Done New false 2342 Acked 3341 Write 2 Done Done New false 2343 Acked 3342 Write 4 Done Done New false 2344 Acked 3343 Write 8 Done Done New false 2345 Acked 3344 Write 10 Done Done New false 2346 Acked 3345 Write 4 Done Done New false 2347 Acked 3346 Flush 0 Done Done New false 2348 Acked 3347 Read 2 Done Done New false 2349 Acked 3348 Read 1 Done Done New false 2350 Acked 3349 Read 10 Done Done New false 2351 Acked 3350 Write 10 Done Done New false 2352 Acked 3351 Read 3 Done Done New false 2353 Acked 3352 Write 6 Done Done New false 2354 Acked 3353 Flush 0 Done Done New false 2355 Acked 3354 Write 5 Done Done New false 2356 Acked 3355 Read 8 Done Done New false 2357 Acked 3356 Read 3 Done Done New false 2358 Acked 3357 Read 6 Done Done New false 2359 Acked 3358 Read 1 Done Done New false 2360 Acked 3359 Write 10 Done Done New false 2361 Acked 3360 Read 3 Done Done New false 2362 Acked 3361 Write 2 Done Done New false 2363 Acked 3362 Write 5 Done Done New false 2364 Acked 3363 Write 8 Done Done New false 2365 Acked 3364 Write 1 Done Done New false 2366 Acked 3365 Write 2 Done Done New false 2367 Acked 3366 Write 9 Done Done New false 2368 Acked 3367 Flush 0 Done Done New false 2369 Acked 3368 Write 3 Done Done New false 2370 Acked 3369 Flush 0 Done Done New false 2371 Acked 3370 Read 9 Done Done New false 2372 Acked 3371 Read 3 Done Done New false 2373 Acked 3372 Read 7 Done Done New false 2374 Acked 3373 Read 10 Done Done New false 2375 Acked 3374 Flush 0 Done Done New false 2376 Acked 3375 Read 5 Done Done New false 2377 Acked 3376 Flush 0 Done Done New false 2378 Acked 3377 Flush 0 Done Done New false 2379 Acked 3378 Write 9 Done Done New false 2380 Acked 3379 Read 9 Done Done New false 2381 Acked 3380 Read 9 Done Done New false 2382 Acked 3381 Write 5 Done Done New false 2383 Acked 3382 Read 6 Done Done New false 2384 Acked 3383 Write 10 Done Done New false 2385 Acked 3384 Flush 0 Done Done New false 2386 Acked 3385 Read 1 Done Done New false 2387 Acked 3386 Read 6 Done Done New false 2388 Acked 3387 Read 8 Done Done New false 2389 Acked 3388 Read 6 Done Done New false 2390 Acked 3389 Read 10 Done Done New false 2391 Acked 3390 Write 6 Done Done New false 2392 Acked 3391 Read 9 Done Done New false 2393 Acked 3392 Flush 0 Done Done New false 2394 Acked 3393 Write 10 Done Done New false 2395 Acked 3394 Read 6 Done Done New false 2396 Acked 3395 Read 1 Done Done New false 2397 Acked 3396 Read 9 Done Done New false 2398 Acked 3397 Read 5 Done Done New false 2399 Acked 3398 Read 4 Done Done New false 2400 Acked 3399 Read 9 Done Done New false 2401 Acked 3400 Write 9 Done Done New false 2402 Acked 3401 Flush 0 Done Done New false 2403 Acked 3402 Read 9 Done Done New false 2404 Acked 3403 Read 10 Done Done New false 2405 Acked 3404 Read 4 Done Done New false 2406 Acked 3405 Write 2 Done Done New false 2407 Acked 3406 Read 4 Done Done New false 2408 Acked 3407 Flush 0 Done Done New false 2409 Acked 3408 Read 1 Done Done New false 2410 Acked 3409 Read 7 Done Done New false 2411 Acked 3410 Flush 0 Done Done New false 2412 Acked 3411 Write 2 Done Done New false 2413 Acked 3412 Read 6 Done Done New false 2414 Acked 3413 Write 8 Done Done New false 2415 Acked 3414 Write 10 Done Done New false 2416 Acked 3415 Flush 0 Done Done New false 2417 Acked 3416 Read 1 Done Done New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 302 302 Sent 0 0 0 0 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3415 3415 3114 Downstairs last five completed: 3114 3113 3112 3111 3110 Upstairs last five completed: 2417 2416 2415 2414 2413 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:57.982388213Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:57.982452798Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:57.982481538Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:57.982508011Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:57.982528322Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:57.982550811Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:57.982743813Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:58.983760574Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:58.983828678Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:58.98385656Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:31:58.983883152Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:58.983903501Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 303 jobs since flush 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:58.983925895Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:58.984164764Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1193) [8830][2] reports Running {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.986034528Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.986141114Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.988274964Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.988307104Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.988330439Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 3114","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.98868314Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.988714199Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.988737312Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.988759612Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:31:59.98878168Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 3417 3416 3415 3414 3413 Upstairs last five completed: 2418 2417 2416 2415 2414 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(1) stop 1 Jan 02 22:32:01.970 INFO request completed, latency_us: 460, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 7dbf0a93-df74-4732-ab2d-60a2cc0d25fd, remote_addr: 127.0.0.1[8820] Got stop action so:false kr:false :37434, local_addr: 127.0.0.1:9998 Jan 02 22:32:01.971 INFO request completed, latency_us: 187, response_code: 200, uri: /state/cid/1, method: GET, req_id: a9fc65bf-a4c5-4177-bcd9-37e0121b1b16, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:01.971769353Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:01.971951406Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:01.972011075Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:01.972057899Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:01.972117812Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:01.97216556Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:02.974081926Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:02.974152179Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:02.974180486Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:02.974207055Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:02.974227544Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:02.974258446Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:02.974301631Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:03.975352378Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:03.975421558Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:03.975453514Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:03.975480607Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:03.975501121Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:03.975524965Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:03.975625765Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:04.976597032Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:04.976664316Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:04.976692301Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:04.976718595Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:04.976742908Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:04.97680571Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:04.976833381Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:32:05.974 INFO request completed, latency_us: 278, response_code: 200, uri: /state/cid/1, method: GET, req_id: c94542f1-19ee-4adc-8fa5-f0b25bbc841c, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 4143 len 28672 data: 3 4 5 4 3 3 3 002/300 Read block 2727 len 40960 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:05.977289969Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:05.977387354Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:05.977440724Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:05.977493099Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:05.977533385Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 2 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:05.977586191Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:05.977632418Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 2129 len 12288 004/300 Write block 3310 len 8192 data: 3 3 005/300 Write block 1861 len 28672 data: 2 2 2 2 2 3 3 006/300 Write block 2411 len 20480 data: 4 5 6 6 5 007/300 Read block 3737 len 28672 008/300 Flush 009/300 Read block 1713 len 8192 010/300 Write block 233 len 4096 data: 4 011/300 Write block 2189 len 16384 data: 3 2 2 2 012/300 Write block 819 len 12288 data: 3 2 2 013/300 Read block 1697 len 36864 014/300 Read block 3116 len 20480 015/300 Read block 4405 len 36864 016/300 Write block 4608 len 28672 data: 3 4 4 4 3 3 3 017/300 Read block 2457 len 4096 018/300 Write block 4228 len 36864 data: 4 4 4 4 4 4 4 3 3 019/300 Read block 1388 len 12288 020/300 Read block 4920 len 12288 021/300 Flush 022/300 Write block 1585 len 4096 data: 3 023/300 Read block 4279 len 32768 024/300 Write block 332 len 40960 data: 4 4 4 2 3 3 4 4 4 5 025/300 Write block 2979 len 36864 data: 3 2 2 2 2 2 3 3 3 026/300 Flush 027/300 Write block 3017 len 12288 data: 5 6 5 028/300 Flush 029/300 Read block 2463 len 40960 030/300 Read block 4144 len 8192 031/300 Write block 103 len 32768 data: 3 3 4 4 2 2 2 2 032/300 Read block 4747 len 28672 033/300 Read block 1252 len 24576 034/300 Write block 2324 len 36864 data: 2 2 3 3 3 4 3 3 3 035/300 Read block 3865 len 32768 036/300 Read block 3402 len 40960 037/300 Read block 16 len 8192 038/300 Read block 2762 len 28672 039/300 Write block 331 len 8192 data: 3 5 040/300 Write block 1072 len 28672 data: 4 4 3 3 3 4 3 041/300 Write block 1176 len 4096 data: 2 042/300 Read block 3303 len 32768 043/300 Write block 2164 len 16384 data: 4 4 3 3 044/300 Read block 3493 len 20480 045/300 Read block 585 len 4096 046/300 Read block 4004 len 40960 047/300 Write block 3397 len 4096 data: 2 048/300 Read block 4407 len 36864 049/300 Write block 3549 len 32768 data: 2 2 2 2 2 2 2 2 050/300 Write block 1369 len 16384 data: 2 2 3 3 051/300 Flush 052/300 Read block 3064 len 8192 053/300 Write block 2406 len 16384 data: 3 3 3 3 054/300 Write block 3396 len 36864 data: 2 3 2 2 2 2 2 2 2 055/300 Write block 2129 len 8192 data: 2 2 056/300 Read block 1004 len 20480 057/300 Flush 058/300 Write block 261 len 32768 data: 4 3 2 2 2 2 4 5 059/300 Read block 4448 len 16384 060/300 Read block 4280 len 24576 061/300 Read block 3689 len 8192 062/300 Write block 25 len 4096 data: 4 063/300 Read block 4813 len 40960 064/300 Read block 3283 len 16384 065/300 Read block 1226 len 20480 066/300 Read block 3798 len 16384 067/300 Read block 2767 len 28672 068/300 Read block 4877 len 28672 069/300 Write block 987 len 36864 data: 3 2 2 2 2 2 2 2 2 070/300 Read block 3544 len 24576 071/300 Write block 1110 len 28672 data: 5 5 5 3 3 3 2 072/300 Read block 2971 len 36864 073/300 Write block 3094 len 8192 data: 2 3 074/300 Read block 3851 len 8192 075/300 Read block 3126 len 24576 076/300 Read block 30 len 32768 077/300 Write block 1934 len 24576 data: 2 2 2 2 2 2 078/300 Read block 937 len 12288 079/300 Read block 570 len 8192 080/300 Read block 4721 len 32768 081/300 Write block 880 len 24576 data: 2 2 2 3 3 3 082/300 Write block 2962 len 36864 data: 4 3 4 4 3 3 3 3 2 083/300 Read block 383 len 24576 084/300 Write block 3451 len 20480 data: 2 2 2 2 3 085/300 Flush 086/300 Read block 4979 len 4096 087/300 Write block 3958 len 4096 data: 4 088/300 Read block 1120 len 20480 089/300 Read block 1658 len 32768 090/300 Write block 1153 len 20480 data: 2 2 2 3 3 091/300 Write block 4441 len 32768 data: 3 4 4 4 3 5 5 5 092/300 Flush 093/300 Flush 094/300 Read block 2778 len 32768 095/300 Write block 233 len 24576 data: 5 4 3 3 2 4 096/300 Read block 2964 len 24576 097/300 Write block 1625 len 28672 data: 4 3 3 2 3 3 3 098/300 Read block 107 len 20480 099/300 Read block 3471 len 28672 100/300 Write block 1134 len 40960 data: 5 5 5 5 4 2 2 4 3 2 101/300 Read block 4017 len 4096 102/300 Write block 1200 len 20480 data: 2 2 2 2 3 103/300 Read block 1819 len 24576 104/300 Read block 2121 len 24576 105/300 Write block 2584 len 8192 data: 4 3 106/300 Read block 3202 len 36864 107/300 Read block 4204 len 28672 108/300 Flush 109/300 Read block 4602 len 16384 110/300 Write block 657 len 12288 data: 4 4 4 111/300 Read block 1407 len 40960 112/300 Read block 3344 len 20480 113/300 Write block 660 len 4096 data: 4 114/300 Read block 1734 len 28672 115/300 Write block 2889 len 40960 data: 3 3 2 2 2 2 2 3 3 4 116/300 Write block 4715 len 24576 data: 2 2 2 2 2 2 117/300 Write block 2880 len 8192 data: 2 2 118/300 Read block 2024 len 4096 119/300 Read block 2594 len 12288 120/300 Flush 121/300 Write block 2795 len 4096 data: 2 122/300 Write block 3048 len 20480 data: 5 3 4 4 4 123/300 Write block 271 len 32768 data: 5 5 5 4 3 2 2 2 124/300 Flush 125/300 Write block 314 len 4096 data: 4 126/300 Read block 4177 len 16384 127/300 Write block 3451 len 36864 data: 3 3 3 3 4 3 3 3 3 128/300 Write block 39 len 16384 data: 3 3 3 3 129/300 Flush 130/300 Read block 2865 len 40960 131/300 Write block 4756 len 28672 data: 3 3 2 2 2 3 3 132/300 Write block 4957 len 8192 data: 4 4 133/300 Flush 134/300 Write block 2155 len 32768 data: 3 3 3 4 3 4 3 3 135/300 Write block 2595 len 20480 data: 2 2 2 2 2 136/300 Write block 2573 len 24576 data: 4 4 4 4 4 4 137/300 Write block 1885 len 8192 data: 2 2 138/300 Flush 139/300 Read block 3240 len 36864 140/300 Read block 4028 len 28672 141/300 Read block 2525 len 4096 142/300 Read block 2829 len 40960 143/300 Flush 144/300 Read block 3469 len 28672 145/300 Read block 2318 len 4096 146/300 Write block 898 len 12288 data: 5 4 4 147/300 Flush 148/300 Write block 4096 len 4096 data: 5 149/300 Write block 3060 len 4096 data: 4 150/300 Read block 3488 len 28672 151/300 Read block 737 len 12288 152/300 Read block 2901 len 16384 153/300 Write block 4511 len 12288 data: 2 2 2 154/300 Read block 406 len 12288 155/300 Read block 2587 len 32768 156/300 Flush 157/300 Write block 3147 len 16384 data: 3 3 4 4 158/300 Read block 782 len 20480 159/300 Flush 160/300 Flush 161/300 Read block 924 len 36864 162/300 Write block 309 len 36864 data: 4 5 5 5 6 5 4 3 2 163/300 Write block 866 len 12288 data: 5 5 4 164/300 Write block 3777 len 28672 data: 2 3 3 3 2 2 2 165/300 Read block 2592 len 20480 166/300 Write block 2342 len 24576 data: 2 2 2 2 2 2 167/300 Read block 1149 len 32768 168/300 Flush 169/300 Read block 240 len 36864 170/300 Read block 277 len 8192 171/300 Write block 683 len 36864 data: 3 3 3 3 4 4 4 4 4 172/300 Read block 3294 len 28672 173/300 Read block 1450 len 28672 174/300 Read block 2823 len 8192 175/300 Flush 176/300 Write block 4730 len 8192 data: 6 6 177/300 Write block 4260 len 40960 data: 3 3 3 3 3 3 5 4 5 5 178/300 Write block 1241 len 36864 data: 3 3 4 4 4 4 3 3 3 179/300 Read block 608 len 36864 180/300 Read block 1937 len 8192 181/300 Read block 4119 len 40960 182/300 Write block 513 len 16384 data: 2 3 3 4 183/300 Read block 4758 len 4096 184/300 Flush 185/300 Read block 388 len 32768 186/300 Read block 3750 len 24576 187/300 Read block 4639 len 24576 188/300 Read block 1412 len 36864 189/300 Read block 1487 len 20480 190/300 Read block 1364 len 28672 191/300 Flush 192/300 Read block 2459 len 8192 193/300 Read block 1266 len 8192 194/300 Read block 1819 len 32768 195/300 Write block 4918 len 36864 data: 4 4 3 3 2 3 3 4 4 196/300 Flush 197/300 Write block 4243 len 32768 data: 2 3 3 3 3 4 4 4 198/300 Write block 3378 len 16384 data: 4 3 2 2 199/300 Write block 2053 len 8192 data: 2 2 200/300 Read block 1996 len 36864 201/300 Write block 146 len 16384 data: 3 3 4 4 202/300 Read block 1131 len 20480 203/300 Read block 2419 len 20480 204/300 Read block 652 len 20480 205/300 Read block 45 len 12288 206/300 Write block 2098 len 32768 data: 2 2 2 2 2 2 2 2 207/300 Write block 2235 len 28672 data: 4 4 4 4 4 5 3 208/300 Read block 2629 len 16384 209/300 Read block 3579 len 12288 210/300 Write block 505 len 20480 data: 2 2 2 2 2 211/300 Read block 2616 len 8192 212/300 Write block 801 len 32768 data: 3 3 3 2 2 2 2 3 213/300 Read block 3829 len 20480 214/300 Write block 1887 len 12288 data: 2 3 2 215/300 Read block 219 len 36864 216/300 Write block 1491 len 20480 data: 3 3 3 3 4 217/300 Read block 2606 len 28672 218/300 Write block 3152 len 12288 data: 4 4 4 219/300 Write block 1762 len 4096 data: 4 220/300 Write block 208 len 40960 data: 3 3 2 2 2 3 3 3 3 3 221/300 Read block 2098 len 4096 222/300 Read block 4046 len 4096 223/300 Read block 3078 len 28672 224/300 Flush 225/300 Read block 2080 len 8192 226/300 Read block 2288 len 32768 227/300 Read block 1474 len 20480 228/300 Read block 861 len 12288 229/300 Flush 230/300 Read block 1368 len 12288 231/300 Write block 1112 len 16384 data: 6 4 4 4 232/300 Read block 1415 len 20480 233/300 Flush 234/300 Write block 3537 len 36864 data: 4 4 4 4 5 5 5 4 3 235/300 Read block 4934 len 12288 236/300 Write block 4693 len 28672 data: 3 3 4 4 4 4 4 237/300 Read block 4549 len 12288 238/300 Write block 3238 len 16384 data: 4 4 5 5 239/300 Write block 708 len 16384 data: 4 4 4 4 240/300 Read block 2303 len 36864 241/300 Flush 242/300 Read block 4530 len 4096 243/300 Read block 155 len 8192 244/300 Write block 4179 len 24576 data: 4 3 3 3 3 2 245/300 Write block 2342 len 24576 data: 3 3 3 3 3 3 246/300 Read block 3801 len 36864 247/300 Write block 3476 len 16384 data: 2 2 2 2 248/300 Read block 334 len 16384 249/300 Write block 1888 len 40960 data: 4 3 3 2 2 2 2 2 2 2 250/300 Write block 437 len 16384 data: 6 7 7 7 251/300 Write block 2385 len 36864 data: 2 2 2 3 3 3 3 3 3 252/300 Write block 4278 len 40960 data: 2 2 2 2 2 2 2 2 2 2 253/300 Write block 3908 len 28672 data: 3 3 3 3 3 3 5 254/300 Write block 82 len 28672 data: 2 3 3 3 3 3 3 255/300 Read block 4362 len 28672 256/300 Read block 4688 len 20480 257/300 Write block 2627 len 36864 data: 4 4 3 2 2 2 2 2 2 258/300 Read block 584 len 28672 259/300 Read block 4688 len 4096 260/300 Read block 4503 len 4096 261/300 Write block 4124 len 20480 data: 3 3 3 3 3 262/300 Read block 753 len 40960 263/300 Read block 4127 len 24576 264/300 Write block 2304 len 12288 data: 4 2 2 265/300 Write block 3663 len 32768 data: 3 2 2 3 4 4 3 3 266/300 Read block 1486 len 36864 267/300 Write block 3379 len 16384 data: 4 3 3 2 268/300 Read block 1305 len 36864 269/300 Write block 4923 len 36864 data: 4 4 5 5 5 5 3 3 4 270/300 Read block 1699 len 32768 271/300 Read block 1375 len 24576 272/300 Read block 385 len 12288 273/300 Write block 698 len 28672 data: 3 3 3 3 3 2 2 274/300 Read block 4690 len 24576 275/300 Write block 2110 len 40960 data: 2 2 2 2 2 2 2 2 2 3 276/300 Flush {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:06.978364235Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:06.978457461Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:06.9784859Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:06.978510943Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:06.978537092Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:06.978557535Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 276 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:06.978579766Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:06.978771421Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 277/300 Write block 4682 len 20480 data: 2 2 2 2 2 278/300 Flush 279/300 Read block 994 len 36864 280/300 Write block 418 len 20480 data: 3 3 3 3 3 281/300 Read block 3704 len 8192 282/300 Read block 2895 len 24576 283/300 Write block 1581 len 24576 data: 2 2 2 2 4 3 284/300 Write block 334 len 28672 data: 5 3 4 4 5 5 5 285/300 Write block 1329 len 24576 data: 4 4 4 3 4 4 286/300 Read block 1665 len 40960 287/300 Write block 204 len 24576 data: 2 3 3 3 4 4 288/300 Read block 1409 len 32768 289/300 Flush 290/300 Read block 1199 len 20480 291/300 Read block 1514 len 24576 292/300 Read block 2030 len 12288 293/300 Write block 4546 len 12288 data: 3 2 2 294/300 Read block 3154 len 16384 295/300 Write block 525 len 32768 data: 3 3 3 3 3 3 2 2 296/300 Write block 2685 len 40960 data: 3 3 4 4 3 3 3 3 3 3 297/300 Read block 3686 len 24576 298/300 Read block 202 len 32768 299/300 Write block 3409 len 40960 data: 3 3 2 2 2 2 2 3 4 4 300/300 Write block 708 len 20480 data: 5 5 5 5 4 Main task has work to do, go find it got dsc Start(1) start 1 Jan 02 22:32:07.079 INFO request completed, latency_us: 339, response_code: 204, uri: /start/cid/1, method: POST, req_id: b2e19d1a-c59a-4797-baeb-d8f5312890b3, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2419 Acked 3418 Write 7 Done New Done true 2420 Acked 3419 Read 10 Done New Done true 2421 Acked 3420 Read 3 Done New Done true 2422 Acked 3421 Write 2 Done New Done true 2423 Acked 3422 Write 7 Done New Done true 2424 Acked 3423 Write 5 Done New Done true 2425 Acked 3424 Read 7 Done New Done true 2426 Acked 3425 Flush 0 Done New Done true 2427 Acked 3426 Read 2 Done New Done true 2428 Acked 3427 Write 1 Done New Done true 2429 Acked 3428 Write 4 Done New Done true 2430 Acked 3429 Write 3 Done New Done true 2431 Acked 3430 Read 9 Done New Done true 2432 Acked 3431 Read 5 Done New Done true 2433 Acked 3432 Read 9 Done New Done true 2434 Acked 3433 Write 7 Done New Done true 2435 Acked 3434 Read 1 Done New Done true 2436 Acked 3435 Write 9 Done New Done true 2437 Acked 3436 Read 3 Done New Done true 2438 Acked 3437 Read 3 Done New Done true 2439 Acked 3438 Flush 0 Done New Done true 2440 Acked 3439 Write 1 Done New Done true 2441 Acked 3440 Read 8 Done New Done true 2442 Acked 3441 Write 10 Done New Done true 2443 Acked 3442 Write 9 Done New Done true 2444 Acked 3443 Flush 0 Done New Done true 2445 Acked 3444 Write 3 Done New Done true 2446 Acked 3445 Flush 0 Done New Done true 2447 Acked 3446 Read 10 Done New Done true 2448 Acked 3447 Read 2 Done New Done true 2449 Acked 3448 Write 8 Done New Done true 2450 Acked 3449 Read 7 Done New Done true 2451 Acked 3450 Read 6 Done New Done true 2452 Acked 3451 Write 9 Done New Done true 2453 Acked 3452 Read 8 Done New Done true 2454 Acked 3453 Read 10 Done New Done true 2455 Acked 3454 Read 2 Done New Done true 2456 Acked 3455 Read 7 Done New Done true 2457 Acked 3456 Write 2 Done New Done true 2458 Acked 3457 Write 7 Done New Done true 2459 Acked 3458 Write 1 Done New Done true 2460 Acked 3459 Read 8 Done New Done true 2461 Acked 3460 Write 4 Done New Done true 2462 Acked 3461 Read 5 Done New Done true 2463 Acked 3462 Read 1 Done New Done true 2464 Acked 3463 Read 10 Done New Done true 2465 Acked 3464 Write 1 Done New Done true 2466 Acked 3465 Read 9 Done New Done true 2467 Acked 3466 Write 8 Done New Done true 2468 Acked 3467 Write 4 Done New Done true 2469 Acked 3468 Flush 0 Done New Done true 2470 Acked 3469 Read 2 Done New Done true 2471 Acked 3470 Write 4 Done New Done true 2472 Acked 3471 Write 9 Done New Done true 2473 Acked 3472 Write 2 Done New Done true 2474 Acked 3473 Read 5 Done New Done true 2475 Acked 3474 Flush 0 Done New Done true 2476 Acked 3475 Write 8 Done New Done true 2477 Acked 3476 Read 4 Done New Done true 2478 Acked 3477 Read 6 Done New Done true 2479 Acked 3478 Read 2 Done New Done true 2480 Acked 3479 Write 1 Done New Done true 2481 Acked 3480 Read 10 Done New Done true 2482 Acked 3481 Read 4 Done New Done true 2483 Acked 3482 Read 5 Done New Done true 2484 Acked 3483 Read 4 Done New Done true 2485 Acked 3484 Read 7 Done New Done true 2486 Acked 3485 Read 7 Done New Done true 2487 Acked 3486 Write 9 Done New Done true 2488 Acked 3487 Read 6 Done New Done true 2489 Acked 3488 Write 7 Done New Done true 2490 Acked 3489 Read 9 Done New Done true 2491 Acked 3490 Write 2 Done New Done true 2492 Acked 3491 Read 2 Done New Done true 2493 Acked 3492 Read 6 Done New Done true 2494 Acked 3493 Read 8 Done New Done true 2495 Acked 3494 Write 6 Done New Done true 2496 Acked 3495 Read 3 Done New Done true 2497 Acked 3496 Read 2 Done New Done true 2498 Acked 3497 Read 8 Done New Done true 2499 Acked 3498 Write 6 Done New Done true 2500 Acked 3499 Write 9 Done New Done true 2501 Acked 3500 Read 6 Done New Done true 2502 Acked 3501 Write 5 Done New Done true 2503 Acked 3502 Flush 0 Done New Done true 2504 Acked 3503 Read 1 Done New Done true 2505 Acked 3504 Write 1 Done New Done true 2506 Acked 3505 Read 5 Done New Done true 2507 Acked 3506 Read 8 Done New Done true 2508 Acked 3507 Write 5 Done New Done true 2509 Acked 3508 Write 8 Done New Done true 2510 Acked 3509 Flush 0 Done New Done true 2511 Acked 3510 Flush 0 Done New Done true 2512 Acked 3511 Read 8 Done New Done true 2513 Acked 3512 Write 6 Done New Done true 2514 Acked 3513 Read 6 Done New Done true 2515 Acked 3514 Write 7 Done New Done true 2516 Acked 3515 Read 5 Done New Done true 2517 Acked 3516 Read 7 Done New Done true 2518 Acked 3517 Write 10 Done New Done true 2519 Acked 3518 Read 1 Done New Done true 2520 Acked 3519 Write 5 Done New Done true 2521 Acked 3520 Read 6 Done New Done true 2522 Acked 3521 Read 6 Done New Done true 2523 Acked 3522 Write 2 Done New Done true 2524 Acked 3523 Read 9 Done New Done true 2525 Acked 3524 Read 7 Done New Done true 2526 Acked 3525 Flush 0 Done New Done true 2527 Acked 3526 Read 4 Done New Done true 2528 Acked 3527 Write 3 Done New Done true 2529 Acked 3528 Read 10 Done New Done true 2530 Acked 3529 Read 5 Done New Done true 2531 Acked 3530 Write 1 Done New Done true 2532 Acked 3531 Read 7 Done New Done true 2533 Acked 3532 Write 10 Done New Done true 2534 Acked 3533 Write 6 Done New Done true 2535 Acked 3534 Write 2 Done New Done true 2536 Acked 3535 Read 1 Done New Done true 2537 Acked 3536 Read 3 Done New Done true 2538 Acked 3537 Flush 0 Done New Done true 2539 Acked 3538 Write 1 Done New Done true 2540 Acked 3539 Write 5 Done New Done true 2541 Acked 3540 Write 8 Done New Done true 2542 Acked 3541 Flush 0 Done New Done true 2543 Acked 3542 Write 1 Done New Done true 2544 Acked 3543 Read 4 Done New Done true 2545 Acked 3544 Write 9 Done New Done true 2546 Acked 3545 Write 4 Done New Done true 2547 Acked 3546 Flush 0 Done New Done true 2548 Acked 3547 Read 10 Done New Done true 2549 Acked 3548 Write 7 Done New Done true 2550 Acked 3549 Write 2 Done New Done true 2551 Acked 3550 Flush 0 Done New Done true 2552 Acked 3551 Write 8 Done New Done true 2553 Acked 3552 Write 5 Done New Done true 2554 Acked 3553 Write 6 Done New Done true 2555 Acked 3554 Write 2 Done New Done true 2556 Acked 3555 Flush 0 Done New Done true 2557 Acked 3556 Read 9 Done New Done true 2558 Acked 3557 Read 7 Done New Done true 2559 Acked 3558 Read 1 Done New Done true 2560 Acked 3559 Read 10 Done New Done true 2561 Acked 3560 Flush 0 Done New Done true 2562 Acked 3561 Read 7 Done New Done true 2563 Acked 3562 Read 1 Done New Done true 2564 Acked 3563 Write 3 Done New Done true 2565 Acked 3564 Flush 0 Done New Done true 2566 Acked 3565 Write 1 Done New Done true 2567 Acked 3566 Write 1 Done New Done true 2568 Acked 3567 Read 7 Done New Done true 2569 Acked 3568 Read 3 Done New Done true 2570 Acked 3569 Read 4 Done New Done true 2571 Acked 3570 Write 3 Done New Done true 2572 Acked 3571 Read 3 Done New Done true 2573 Acked 3572 Read 8 Done New Done true 2574 Acked 3573 Flush 0 Done New Done true 2575 Acked 3574 Write 4 Done New Done true 2576 Acked 3575 Read 5 Done New Done true 2577 Acked 3576 Flush 0 Done New Done true 2578 Acked 3577 Flush 0 Done New Done true 2579 Acked 3578 Read 9 Done New Done true 2580 Acked 3579 Write 9 Done New Done true 2581 Acked 3580 Write 3 Done New Done true 2582 Acked 3581 Write 7 Done New Done true 2583 Acked 3582 Read 5 Done New Done true 2584 Acked 3583 Write 6 Done New Done true 2585 Acked 3584 Read 8 Done New Done true 2586 Acked 3585 Flush 0 Done New Done true 2587 Acked 3586 Read 9 Done New Done true 2588 Acked 3587 Read 2 Done New Done true 2589 Acked 3588 Write 9 Done New Done true 2590 Acked 3589 Read 7 Done New Done true 2591 Acked 3590 Read 7 Done New Done true 2592 Acked 3591 Read 2 Done New Done true 2593 Acked 3592 Flush 0 Done New Done true 2594 Acked 3593 Write 2 Done New Done true 2595 Acked 3594 Write 10 Done New Done true 2596 Acked 3595 Write 9 Done New Done true 2597 Acked 3596 Read 9 Done New Done true 2598 Acked 3597 Read 2 Done New Done true 2599 Acked 3598 Read 10 Done New Done true 2600 Acked 3599 Write 4 Done New Done true 2601 Acked 3600 Read 1 Done New Done true 2602 Acked 3601 Flush 0 Done New Done true 2603 Acked 3602 Read 8 Done New Done true 2604 Acked 3603 Read 6 Done New Done true 2605 Acked 3604 Read 6 Done New Done true 2606 Acked 3605 Read 9 Done New Done true 2607 Acked 3606 Read 5 Done New Done true 2608 Acked 3607 Read 7 Done New Done true 2609 Acked 3608 Flush 0 Done New Done true 2610 Acked 3609 Read 2 Done New Done true 2611 Acked 3610 Read 2 Done New Done true 2612 Acked 3611 Read 8 Done New Done true 2613 Acked 3612 Write 9 Done New Done true 2614 Acked 3613 Flush 0 Done New Done true 2615 Acked 3614 Write 8 Done New Done true 2616 Acked 3615 Write 4 Done New Done true 2617 Acked 3616 Write 2 Done New Done true 2618 Acked 3617 Read 9 Done New Done true 2619 Acked 3618 Write 4 Done New Done true 2620 Acked 3619 Read 5 Done New Done true 2621 Acked 3620 Read 5 Done New Done true 2622 Acked 3621 Read 5 Done New Done true 2623 Acked 3622 Read 3 Done New Done true 2624 Acked 3623 Write 8 Done New Done true 2625 Acked 3624 Write 7 Done New Done true 2626 Acked 3625 Read 4 Done New Done true 2627 Acked 3626 Read 3 Done New Done true 2628 Acked 3627 Write 5 Done New Done true 2629 Acked 3628 Read 2 Done New Done true 2630 Acked 3629 Write 8 Done New Done true 2631 Acked 3630 Read 5 Done New Done true 2632 Acked 3631 Write 3 Done New Done true 2633 Acked 3632 Read 9 Done New Done true 2634 Acked 3633 Write 5 Done New Done true 2635 Acked 3634 Read 7 Done New Done true 2636 Acked 3635 Write 3 Done New Done true 2637 Acked 3636 Write 1 Done New Done true 2638 Acked 3637 Write 10 Done New Done true 2639 Acked 3638 Read 1 Done New Done true 2640 Acked 3639 Read 1 Done New Done true 2641 Acked 3640 Read 7 Done New Done true 2642 Acked 3641 Flush 0 Done New Done true 2643 Acked 3642 Read 2 Done New Done true 2644 Acked 3643 Read 8 Done New Done true 2645 Acked 3644 Read 5 Done New Done true 2646 Acked 3645 Read 3 Done New Done true 2647 Acked 3646 Flush 0 Done New Done true 2648 Acked 3647 Read 3 Done New Done true 2649 Acked 3648 Write 4 Done New Done true 2650 Acked 3649 Read 5 Done New Done true 2651 Acked 3650 Flush 0 Done New Done true 2652 Acked 3651 Write 9 Done New Done true 2653 Acked 3652 Read 3 Done New Done true 2654 Acked 3653 Write 7 Done New Done true 2655 Acked 3654 Read 3 Done New Done true 2656 Acked 3655 Write 4 Done New Done true 2657 Acked 3656 Write 4 Done New Done true 2658 Acked 3657 Read 9 Done New Done true 2659 Acked 3658 Flush 0 Done New Done true 2660 Acked 3659 Read 1 Done New Done true 2661 Acked 3660 Read 2 Done New Done true 2662 Acked 3661 Write 6 Done New Done true 2663 Acked 3662 Write 6 Done New Done true 2664 Acked 3663 Read 9 Done New Done true 2665 Acked 3664 Write 4 Done New Done true 2666 Acked 3665 Read 4 Done New Done true 2667 Acked 3666 Write 10 Done New Done true 2668 Acked 3667 Write 4 Done New Done true 2669 Acked 3668 Write 9 Done New Done true 2670 Acked 3669 Write 10 Done New Done true 2671 Acked 3670 Write 7 Done New Done true 2672 Acked 3671 Write 7 Done New Done true 2673 Acked 3672 Read 7 Done New Done true 2674 Acked 3673 Read 5 Done New Done true 2675 Acked 3674 Write 9 Done New Done true 2676 Acked 3675 Read 7 Done New Done true 2677 Acked 3676 Read 1 Done New Done true 2678 Acked 3677 Read 1 Done New Done true 2679 Acked 3678 Write 5 Done New Done true 2680 Acked 3679 Read 10 Done New Done true 2681 Acked 3680 Read 6 Done New Done true 2682 Acked 3681 Write 3 Done New Done true 2683 Acked 3682 Write 8 Done New Done true 2684 Acked 3683 Read 9 Done New Done true 2685 Acked 3684 Write 4 Done New Done true 2686 Acked 3685 Read 9 Done New Done true 2687 Acked 3686 Write 9 Done New Done true 2688 Acked 3687 Read 8 Done New Done true 2689 Acked 3688 Read 6 Done New Done true 2690 Acked 3689 Read 3 Done New Done true 2691 Acked 3690 Write 7 Done New Done true 2692 Acked 3691 Read 6 Done New Done true 2693 Acked 3692 Write 10 Done New Done true 2694 Acked 3693 Flush 0 Done New Done true 2695 Acked 3694 Write 5 Done New Done false 2696 Acked 3695 Flush 0 Done New Done false 2697 Acked 3696 Read 9 Done New Done false 2698 Acked 3697 Write 5 Done New Done false 2699 Acked 3698 Read 2 Done New Done false 2700 Acked 3699 Read 6 Done New Done false 2701 Acked 3700 Write 6 Done New Done false 2702 Acked 3701 Write 7 Done New Done false 2703 Acked 3702 Write 6 Done New Done false 2704 Acked 3703 Read 10 Done New Done false 2705 Acked 3704 Write 6 Done New Done false 2706 Acked 3705 Read 8 Done New Done false 2707 Acked 3706 Flush 0 Done New Done false 2708 Acked 3707 Read 5 Done New Done false 2709 Acked 3708 Read 6 Done New Done false 2710 Acked 3709 Read 3 Done New Done false 2711 Acked 3710 Write 3 Done New Done false 2712 Acked 3711 Read 4 Done New Done false 2713 Acked 3712 Write 8 Done New Done false 2714 Acked 3713 Write 10 Done New Done false 2715 Acked 3714 Read 6 Done New Done false 2716 Acked 3715 Read 8 Done New Done false 2717 Acked 3716 Write 10 Done New Done false 2718 Acked 3717 Write 5 Sent New Sent false STATES DS:0 DS:1 DS:2 TOTAL New 0 300 0 300 Sent 1 0 1 2 Done 299 0 299 598 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3706 3417 3706 Downstairs last five completed: 3417 3416 3415 3414 3413 Upstairs last five completed: 2718 2717 2716 2715 2714 CLIENT: Up:0 ds:300 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:07.979981785Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:07.980063317Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:07.980100943Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:07.980128569Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:07.980148969Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:07.98017186Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:07.980412007Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:08.981516205Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:08.981631681Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:08.981659922Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:08.981686874Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:08.981707549Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:08.981730239Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:08.981968444Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:09.983003978Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:09.983118252Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:09.983193207Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:09.983241718Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:09.983262334Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:09.983284989Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:09.983509499Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:10.98435768Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:10.984426361Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:10.984455297Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:10.984481986Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:10.984502117Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:10.984524475Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:10.984736887Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2419 Acked 3418 Write 7 Done New Done true 2420 Acked 3419 Read 10 Done New Done true 2421 Acked 3420 Read 3 Done New Done true 2422 Acked 3421 Write 2 Done New Done true 2423 Acked 3422 Write 7 Done New Done true 2424 Acked 3423 Write 5 Done New Done true 2425 Acked 3424 Read 7 Done New Done true 2426 Acked 3425 Flush 0 Done New Done true 2427 Acked 3426 Read 2 Done New Done true 2428 Acked 3427 Write 1 Done New Done true 2429 Acked 3428 Write 4 Done New Done true 2430 Acked 3429 Write 3 Done New Done true 2431 Acked 3430 Read 9 Done New Done true 2432 Acked 3431 Read 5 Done New Done true 2433 Acked 3432 Read 9 Done New Done true 2434 Acked 3433 Write 7 Done New Done true 2435 Acked 3434 Read 1 Done New Done true 2436 Acked 3435 Write 9 Done New Done true 2437 Acked 3436 Read 3 Done New Done true 2438 Acked 3437 Read 3 Done New Done true 2439 Acked 3438 Flush 0 Done New Done true 2440 Acked 3439 Write 1 Done New Done true 2441 Acked 3440 Read 8 Done New Done true 2442 Acked 3441 Write 10 Done New Done true 2443 Acked 3442 Write 9 Done New Done true 2444 Acked 3443 Flush 0 Done New Done true 2445 Acked 3444 Write 3 Done New Done true 2446 Acked 3445 Flush 0 Done New Done true 2447 Acked 3446 Read 10 Done New Done true 2448 Acked 3447 Read 2 Done New Done true 2449 Acked 3448 Write 8 Done New Done true 2450 Acked 3449 Read 7 Done New Done true 2451 Acked 3450 Read 6 Done New Done true 2452 Acked 3451 Write 9 Done New Done true 2453 Acked 3452 Read 8 Done New Done true 2454 Acked 3453 Read 10 Done New Done true 2455 Acked 3454 Read 2 Done New Done true 2456 Acked 3455 Read 7 Done New Done true 2457 Acked 3456 Write 2 Done New Done true 2458 Acked 3457 Write 7 Done New Done true 2459 Acked 3458 Write 1 Done New Done true 2460 Acked 3459 Read 8 Done New Done true 2461 Acked 3460 Write 4 Done New Done true 2462 Acked 3461 Read 5 Done New Done true 2463 Acked 3462 Read 1 Done New Done true 2464 Acked 3463 Read 10 Done New Done true 2465 Acked 3464 Write 1 Done New Done true 2466 Acked 3465 Read 9 Done New Done true 2467 Acked 3466 Write 8 Done New Done true 2468 Acked 3467 Write 4 Done New Done true 2469 Acked 3468 Flush 0 Done New Done true 2470 Acked 3469 Read 2 Done New Done true 2471 Acked 3470 Write 4 Done New Done true 2472 Acked 3471 Write 9 Done New Done true 2473 Acked 3472 Write 2 Done New Done true 2474 Acked 3473 Read 5 Done New Done true 2475 Acked 3474 Flush 0 Done New Done true 2476 Acked 3475 Write 8 Done New Done true 2477 Acked 3476 Read 4 Done New Done true 2478 Acked 3477 Read 6 Done New Done true 2479 Acked 3478 Read 2 Done New Done true 2480 Acked 3479 Write 1 Done New Done true 2481 Acked 3480 Read 10 Done New Done true 2482 Acked 3481 Read 4 Done New Done true 2483 Acked 3482 Read 5 Done New Done true 2484 Acked 3483 Read 4 Done New Done true 2485 Acked 3484 Read 7 Done New Done true 2486 Acked 3485 Read 7 Done New Done true 2487 Acked 3486 Write 9 Done New Done true 2488 Acked 3487 Read 6 Done New Done true 2489 Acked 3488 Write 7 Done New Done true 2490 Acked 3489 Read 9 Done New Done true 2491 Acked 3490 Write 2 Done New Done true 2492 Acked 3491 Read 2 Done New Done true 2493 Acked 3492 Read 6 Done New Done true 2494 Acked 3493 Read 8 Done New Done true 2495 Acked 3494 Write 6 Done New Done true 2496 Acked 3495 Read 3 Done New Done true 2497 Acked 3496 Read 2 Done New Done true 2498 Acked 3497 Read 8 Done New Done true 2499 Acked 3498 Write 6 Done New Done true 2500 Acked 3499 Write 9 Done New Done true 2501 Acked 3500 Read 6 Done New Done true 2502 Acked 3501 Write 5 Done New Done true 2503 Acked 3502 Flush 0 Done New Done true 2504 Acked 3503 Read 1 Done New Done true 2505 Acked 3504 Write 1 Done New Done true 2506 Acked 3505 Read 5 Done New Done true 2507 Acked 3506 Read 8 Done New Done true 2508 Acked 3507 Write 5 Done New Done true 2509 Acked 3508 Write 8 Done New Done true 2510 Acked 3509 Flush 0 Done New Done true 2511 Acked 3510 Flush 0 Done New Done true 2512 Acked 3511 Read 8 Done New Done true 2513 Acked 3512 Write 6 Done New Done true 2514 Acked 3513 Read 6 Done New Done true 2515 Acked 3514 Write 7 Done New Done true 2516 Acked 3515 Read 5 Done New Done true 2517 Acked 3516 Read 7 Done New Done true 2518 Acked 3517 Write 10 Done New Done true 2519 Acked 3518 Read 1 Done New Done true 2520 Acked 3519 Write 5 Done New Done true 2521 Acked 3520 Read 6 Done New Done true 2522 Acked 3521 Read 6 Done New Done true 2523 Acked 3522 Write 2 Done New Done true 2524 Acked 3523 Read 9 Done New Done true 2525 Acked 3524 Read 7 Done New Done true 2526 Acked 3525 Flush 0 Done New Done true 2527 Acked 3526 Read 4 Done New Done true 2528 Acked 3527 Write 3 Done New Done true 2529 Acked 3528 Read 10 Done New Done true 2530 Acked 3529 Read 5 Done New Done true 2531 Acked 3530 Write 1 Done New Done true 2532 Acked 3531 Read 7 Done New Done true 2533 Acked 3532 Write 10 Done New Done true 2534 Acked 3533 Write 6 Done New Done true 2535 Acked 3534 Write 2 Done New Done true 2536 Acked 3535 Read 1 Done New Done true 2537 Acked 3536 Read 3 Done New Done true 2538 Acked 3537 Flush 0 Done New Done true 2539 Acked 3538 Write 1 Done New Done true 2540 Acked 3539 Write 5 Done New Done true 2541 Acked 3540 Write 8 Done New Done true 2542 Acked 3541 Flush 0 Done New Done true 2543 Acked 3542 Write 1 Done New Done true 2544 Acked 3543 Read 4 Done New Done true 2545 Acked 3544 Write 9 Done New Done true 2546 Acked 3545 Write 4 Done New Done true 2547 Acked 3546 Flush 0 Done New Done true 2548 Acked 3547 Read 10 Done New Done true 2549 Acked 3548 Write 7 Done New Done true 2550 Acked 3549 Write 2 Done New Done true 2551 Acked 3550 Flush 0 Done New Done true 2552 Acked 3551 Write 8 Done New Done true 2553 Acked 3552 Write 5 Done New Done true 2554 Acked 3553 Write 6 Done New Done true 2555 Acked 3554 Write 2 Done New Done true 2556 Acked 3555 Flush 0 Done New Done true 2557 Acked 3556 Read 9 Done New Done true 2558 Acked 3557 Read 7 Done New Done true 2559 Acked 3558 Read 1 Done New Done true 2560 Acked 3559 Read 10 Done New Done true 2561 Acked 3560 Flush 0 Done New Done true 2562 Acked 3561 Read 7 Done New Done true 2563 Acked 3562 Read 1 Done New Done true 2564 Acked 3563 Write 3 Done New Done true 2565 Acked 3564 Flush 0 Done New Done true 2566 Acked 3565 Write 1 Done New Done true 2567 Acked 3566 Write 1 Done New Done true 2568 Acked 3567 Read 7 Done New Done true 2569 Acked 3568 Read 3 Done New Done true 2570 Acked 3569 Read 4 Done New Done true 2571 Acked 3570 Write 3 Done New Done true 2572 Acked 3571 Read 3 Done New Done true 2573 Acked 3572 Read 8 Done New Done true 2574 Acked 3573 Flush 0 Done New Done true 2575 Acked 3574 Write 4 Done New Done true 2576 Acked 3575 Read 5 Done New Done true 2577 Acked 3576 Flush 0 Done New Done true 2578 Acked 3577 Flush 0 Done New Done true 2579 Acked 3578 Read 9 Done New Done true 2580 Acked 3579 Write 9 Done New Done true 2581 Acked 3580 Write 3 Done New Done true 2582 Acked 3581 Write 7 Done New Done true 2583 Acked 3582 Read 5 Done New Done true 2584 Acked 3583 Write 6 Done New Done true 2585 Acked 3584 Read 8 Done New Done true 2586 Acked 3585 Flush 0 Done New Done true 2587 Acked 3586 Read 9 Done New Done true 2588 Acked 3587 Read 2 Done New Done true 2589 Acked 3588 Write 9 Done New Done true 2590 Acked 3589 Read 7 Done New Done true 2591 Acked 3590 Read 7 Done New Done true 2592 Acked 3591 Read 2 Done New Done true 2593 Acked 3592 Flush 0 Done New Done true 2594 Acked 3593 Write 2 Done New Done true 2595 Acked 3594 Write 10 Done New Done true 2596 Acked 3595 Write 9 Done New Done true 2597 Acked 3596 Read 9 Done New Done true 2598 Acked 3597 Read 2 Done New Done true 2599 Acked 3598 Read 10 Done New Done true 2600 Acked 3599 Write 4 Done New Done true 2601 Acked 3600 Read 1 Done New Done true 2602 Acked 3601 Flush 0 Done New Done true 2603 Acked 3602 Read 8 Done New Done true 2604 Acked 3603 Read 6 Done New Done true 2605 Acked 3604 Read 6 Done New Done true 2606 Acked 3605 Read 9 Done New Done true 2607 Acked 3606 Read 5 Done New Done true 2608 Acked 3607 Read 7 Done New Done true 2609 Acked 3608 Flush 0 Done New Done true 2610 Acked 3609 Read 2 Done New Done true 2611 Acked 3610 Read 2 Done New Done true 2612 Acked 3611 Read 8 Done New Done true 2613 Acked 3612 Write 9 Done New Done true 2614 Acked 3613 Flush 0 Done New Done true 2615 Acked 3614 Write 8 Done New Done true 2616 Acked 3615 Write 4 Done New Done true 2617 Acked 3616 Write 2 Done New Done true 2618 Acked 3617 Read 9 Done New Done true 2619 Acked 3618 Write 4 Done New Done true 2620 Acked 3619 Read 5 Done New Done true 2621 Acked 3620 Read 5 Done New Done true 2622 Acked 3621 Read 5 Done New Done true 2623 Acked 3622 Read 3 Done New Done true 2624 Acked 3623 Write 8 Done New Done true 2625 Acked 3624 Write 7 Done New Done true 2626 Acked 3625 Read 4 Done New Done true 2627 Acked 3626 Read 3 Done New Done true 2628 Acked 3627 Write 5 Done New Done true 2629 Acked 3628 Read 2 Done New Done true 2630 Acked 3629 Write 8 Done New Done true 2631 Acked 3630 Read 5 Done New Done true 2632 Acked 3631 Write 3 Done New Done true 2633 Acked 3632 Read 9 Done New Done true 2634 Acked 3633 Write 5 Done New Done true 2635 Acked 3634 Read 7 Done New Done true 2636 Acked 3635 Write 3 Done New Done true 2637 Acked 3636 Write 1 Done New Done true 2638 Acked 3637 Write 10 Done New Done true 2639 Acked 3638 Read 1 Done New Done true 2640 Acked 3639 Read 1 Done New Done true 2641 Acked 3640 Read 7 Done New Done true 2642 Acked 3641 Flush 0 Done New Done true 2643 Acked 3642 Read 2 Done New Done true 2644 Acked 3643 Read 8 Done New Done true 2645 Acked 3644 Read 5 Done New Done true 2646 Acked 3645 Read 3 Done New Done true 2647 Acked 3646 Flush 0 Done New Done true 2648 Acked 3647 Read 3 Done New Done true 2649 Acked 3648 Write 4 Done New Done true 2650 Acked 3649 Read 5 Done New Done true 2651 Acked 3650 Flush 0 Done New Done true 2652 Acked 3651 Write 9 Done New Done true 2653 Acked 3652 Read 3 Done New Done true 2654 Acked 3653 Write 7 Done New Done true 2655 Acked 3654 Read 3 Done New Done true 2656 Acked 3655 Write 4 Done New Done true 2657 Acked 3656 Write 4 Done New Done true 2658 Acked 3657 Read 9 Done New Done true 2659 Acked 3658 Flush 0 Done New Done true 2660 Acked 3659 Read 1 Done New Done true 2661 Acked 3660 Read 2 Done New Done true 2662 Acked 3661 Write 6 Done New Done true 2663 Acked 3662 Write 6 Done New Done true 2664 Acked 3663 Read 9 Done New Done true 2665 Acked 3664 Write 4 Done New Done true 2666 Acked 3665 Read 4 Done New Done true 2667 Acked 3666 Write 10 Done New Done true 2668 Acked 3667 Write 4 Done New Done true 2669 Acked 3668 Write 9 Done New Done true 2670 Acked 3669 Write 10 Done New Done true 2671 Acked 3670 Write 7 Done New Done true 2672 Acked 3671 Write 7 Done New Done true 2673 Acked 3672 Read 7 Done New Done true 2674 Acked 3673 Read 5 Done New Done true 2675 Acked 3674 Write 9 Done New Done true 2676 Acked 3675 Read 7 Done New Done true 2677 Acked 3676 Read 1 Done New Done true 2678 Acked 3677 Read 1 Done New Done true 2679 Acked 3678 Write 5 Done New Done true 2680 Acked 3679 Read 10 Done New Done true 2681 Acked 3680 Read 6 Done New Done true 2682 Acked 3681 Write 3 Done New Done true 2683 Acked 3682 Write 8 Done New Done true 2684 Acked 3683 Read 9 Done New Done true 2685 Acked 3684 Write 4 Done New Done true 2686 Acked 3685 Read 9 Done New Done true 2687 Acked 3686 Write 9 Done New Done true 2688 Acked 3687 Read 8 Done New Done true 2689 Acked 3688 Read 6 Done New Done true 2690 Acked 3689 Read 3 Done New Done true 2691 Acked 3690 Write 7 Done New Done true 2692 Acked 3691 Read 6 Done New Done true 2693 Acked 3692 Write 10 Done New Done true 2694 Acked 3693 Flush 0 Done New Done true 2695 Acked 3694 Write 5 Done New Done true 2696 Acked 3695 Flush 0 Done New Done true 2697 Acked 3696 Read 9 Done New Done true 2698 Acked 3697 Write 5 Done New Done true 2699 Acked 3698 Read 2 Done New Done true 2700 Acked 3699 Read 6 Done New Done true 2701 Acked 3700 Write 6 Done New Done true 2702 Acked 3701 Write 7 Done New Done true 2703 Acked 3702 Write 6 Done New Done true 2704 Acked 3703 Read 10 Done New Done true 2705 Acked 3704 Write 6 Done New Done true 2706 Acked 3705 Read 8 Done New Done true 2707 Acked 3706 Flush 0 Done New Done true 2708 Acked 3707 Read 5 Done New Done true 2709 Acked 3708 Read 6 Done New Done true 2710 Acked 3709 Read 3 Done New Done true 2711 Acked 3710 Write 3 Done New Done true 2712 Acked 3711 Read 4 Done New Done true 2713 Acked 3712 Write 8 Done New Done true 2714 Acked 3713 Write 10 Done New Done true 2715 Acked 3714 Read 6 Done New Done true 2716 Acked 3715 Read 8 Done New Done true 2717 Acked 3716 Write 10 Done New Done true 2718 Acked 3717 Write 5 Done New Done true 2719 Acked 3718 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 301 0 301 Sent 0 0 0 0 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3718 3417 3718 Downstairs last five completed: 3417 3416 3415 3414 3413 Upstairs last five completed: 2719 2718 2717 2716 2715 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:11.985426237Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:11.985492724Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:11.985520747Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:11.985544855Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:11.985570875Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:11.985602013Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:11.985625262Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:11.985891443Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:12.986840444Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:12.986907517Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:12.986935872Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:12.986962277Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:12.986982688Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:12.987010777Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:12.987248645Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:13.988078941Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:13.988144097Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:13.988172147Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:13.988206315Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:13.988227313Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:13.988249821Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:13.988485804Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:14.989278048Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:14.989348591Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:14.989376621Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:14.989403048Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:14.989423559Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:14.989471696Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:14.989674Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2419 Acked 3418 Write 7 Done New Done true 2420 Acked 3419 Read 10 Done New Done true 2421 Acked 3420 Read 3 Done New Done true 2422 Acked 3421 Write 2 Done New Done true 2423 Acked 3422 Write 7 Done New Done true 2424 Acked 3423 Write 5 Done New Done true 2425 Acked 3424 Read 7 Done New Done true 2426 Acked 3425 Flush 0 Done New Done true 2427 Acked 3426 Read 2 Done New Done true 2428 Acked 3427 Write 1 Done New Done true 2429 Acked 3428 Write 4 Done New Done true 2430 Acked 3429 Write 3 Done New Done true 2431 Acked 3430 Read 9 Done New Done true 2432 Acked 3431 Read 5 Done New Done true 2433 Acked 3432 Read 9 Done New Done true 2434 Acked 3433 Write 7 Done New Done true 2435 Acked 3434 Read 1 Done New Done true 2436 Acked 3435 Write 9 Done New Done true 2437 Acked 3436 Read 3 Done New Done true 2438 Acked 3437 Read 3 Done New Done true 2439 Acked 3438 Flush 0 Done New Done true 2440 Acked 3439 Write 1 Done New Done true 2441 Acked 3440 Read 8 Done New Done true 2442 Acked 3441 Write 10 Done New Done true 2443 Acked 3442 Write 9 Done New Done true 2444 Acked 3443 Flush 0 Done New Done true 2445 Acked 3444 Write 3 Done New Done true 2446 Acked 3445 Flush 0 Done New Done true 2447 Acked 3446 Read 10 Done New Done true 2448 Acked 3447 Read 2 Done New Done true 2449 Acked 3448 Write 8 Done New Done true 2450 Acked 3449 Read 7 Done New Done true 2451 Acked 3450 Read 6 Done New Done true 2452 Acked 3451 Write 9 Done New Done true 2453 Acked 3452 Read 8 Done New Done true 2454 Acked 3453 Read 10 Done New Done true 2455 Acked 3454 Read 2 Done New Done true 2456 Acked 3455 Read 7 Done New Done true 2457 Acked 3456 Write 2 Done New Done true 2458 Acked 3457 Write 7 Done New Done true 2459 Acked 3458 Write 1 Done New Done true 2460 Acked 3459 Read 8 Done New Done true 2461 Acked 3460 Write 4 Done New Done true 2462 Acked 3461 Read 5 Done New Done true 2463 Acked 3462 Read 1 Done New Done true 2464 Acked 3463 Read 10 Done New Done true 2465 Acked 3464 Write 1 Done New Done true 2466 Acked 3465 Read 9 Done New Done true 2467 Acked 3466 Write 8 Done New Done true 2468 Acked 3467 Write 4 Done New Done true 2469 Acked 3468 Flush 0 Done New Done true 2470 Acked 3469 Read 2 Done New Done true 2471 Acked 3470 Write 4 Done New Done true 2472 Acked 3471 Write 9 Done New Done true 2473 Acked 3472 Write 2 Done New Done true 2474 Acked 3473 Read 5 Done New Done true 2475 Acked 3474 Flush 0 Done New Done true 2476 Acked 3475 Write 8 Done New Done true 2477 Acked 3476 Read 4 Done New Done true 2478 Acked 3477 Read 6 Done New Done true 2479 Acked 3478 Read 2 Done New Done true 2480 Acked 3479 Write 1 Done New Done true 2481 Acked 3480 Read 10 Done New Done true 2482 Acked 3481 Read 4 Done New Done true 2483 Acked 3482 Read 5 Done New Done true 2484 Acked 3483 Read 4 Done New Done true 2485 Acked 3484 Read 7 Done New Done true 2486 Acked 3485 Read 7 Done New Done true 2487 Acked 3486 Write 9 Done New Done true 2488 Acked 3487 Read 6 Done New Done true 2489 Acked 3488 Write 7 Done New Done true 2490 Acked 3489 Read 9 Done New Done true 2491 Acked 3490 Write 2 Done New Done true 2492 Acked 3491 Read 2 Done New Done true 2493 Acked 3492 Read 6 Done New Done true 2494 Acked 3493 Read 8 Done New Done true 2495 Acked 3494 Write 6 Done New Done true 2496 Acked 3495 Read 3 Done New Done true 2497 Acked 3496 Read 2 Done New Done true 2498 Acked 3497 Read 8 Done New Done true 2499 Acked 3498 Write 6 Done New Done true 2500 Acked 3499 Write 9 Done New Done true 2501 Acked 3500 Read 6 Done New Done true 2502 Acked 3501 Write 5 Done New Done true 2503 Acked 3502 Flush 0 Done New Done true 2504 Acked 3503 Read 1 Done New Done true 2505 Acked 3504 Write 1 Done New Done true 2506 Acked 3505 Read 5 Done New Done true 2507 Acked 3506 Read 8 Done New Done true 2508 Acked 3507 Write 5 Done New Done true 2509 Acked 3508 Write 8 Done New Done true 2510 Acked 3509 Flush 0 Done New Done true 2511 Acked 3510 Flush 0 Done New Done true 2512 Acked 3511 Read 8 Done New Done true 2513 Acked 3512 Write 6 Done New Done true 2514 Acked 3513 Read 6 Done New Done true 2515 Acked 3514 Write 7 Done New Done true 2516 Acked 3515 Read 5 Done New Done true 2517 Acked 3516 Read 7 Done New Done true 2518 Acked 3517 Write 10 Done New Done true 2519 Acked 3518 Read 1 Done New Done true 2520 Acked 3519 Write 5 Done New Done true 2521 Acked 3520 Read 6 Done New Done true 2522 Acked 3521 Read 6 Done New Done true 2523 Acked 3522 Write 2 Done New Done true 2524 Acked 3523 Read 9 Done New Done true 2525 Acked 3524 Read 7 Done New Done true 2526 Acked 3525 Flush 0 Done New Done true 2527 Acked 3526 Read 4 Done New Done true 2528 Acked 3527 Write 3 Done New Done true 2529 Acked 3528 Read 10 Done New Done true 2530 Acked 3529 Read 5 Done New Done true 2531 Acked 3530 Write 1 Done New Done true 2532 Acked 3531 Read 7 Done New Done true 2533 Acked 3532 Write 10 Done New Done true 2534 Acked 3533 Write 6 Done New Done true 2535 Acked 3534 Write 2 Done New Done true 2536 Acked 3535 Read 1 Done New Done true 2537 Acked 3536 Read 3 Done New Done true 2538 Acked 3537 Flush 0 Done New Done true 2539 Acked 3538 Write 1 Done New Done true 2540 Acked 3539 Write 5 Done New Done true 2541 Acked 3540 Write 8 Done New Done true 2542 Acked 3541 Flush 0 Done New Done true 2543 Acked 3542 Write 1 Done New Done true 2544 Acked 3543 Read 4 Done New Done true 2545 Acked 3544 Write 9 Done New Done true 2546 Acked 3545 Write 4 Done New Done true 2547 Acked 3546 Flush 0 Done New Done true 2548 Acked 3547 Read 10 Done New Done true 2549 Acked 3548 Write 7 Done New Done true 2550 Acked 3549 Write 2 Done New Done true 2551 Acked 3550 Flush 0 Done New Done true 2552 Acked 3551 Write 8 Done New Done true 2553 Acked 3552 Write 5 Done New Done true 2554 Acked 3553 Write 6 Done New Done true 2555 Acked 3554 Write 2 Done New Done true 2556 Acked 3555 Flush 0 Done New Done true 2557 Acked 3556 Read 9 Done New Done true 2558 Acked 3557 Read 7 Done New Done true 2559 Acked 3558 Read 1 Done New Done true 2560 Acked 3559 Read 10 Done New Done true 2561 Acked 3560 Flush 0 Done New Done true 2562 Acked 3561 Read 7 Done New Done true 2563 Acked 3562 Read 1 Done New Done true 2564 Acked 3563 Write 3 Done New Done true 2565 Acked 3564 Flush 0 Done New Done true 2566 Acked 3565 Write 1 Done New Done true 2567 Acked 3566 Write 1 Done New Done true 2568 Acked 3567 Read 7 Done New Done true 2569 Acked 3568 Read 3 Done New Done true 2570 Acked 3569 Read 4 Done New Done true 2571 Acked 3570 Write 3 Done New Done true 2572 Acked 3571 Read 3 Done New Done true 2573 Acked 3572 Read 8 Done New Done true 2574 Acked 3573 Flush 0 Done New Done true 2575 Acked 3574 Write 4 Done New Done true 2576 Acked 3575 Read 5 Done New Done true 2577 Acked 3576 Flush 0 Done New Done true 2578 Acked 3577 Flush 0 Done New Done true 2579 Acked 3578 Read 9 Done New Done true 2580 Acked 3579 Write 9 Done New Done true 2581 Acked 3580 Write 3 Done New Done true 2582 Acked 3581 Write 7 Done New Done true 2583 Acked 3582 Read 5 Done New Done true 2584 Acked 3583 Write 6 Done New Done true 2585 Acked 3584 Read 8 Done New Done true 2586 Acked 3585 Flush 0 Done New Done true 2587 Acked 3586 Read 9 Done New Done true 2588 Acked 3587 Read 2 Done New Done true 2589 Acked 3588 Write 9 Done New Done true 2590 Acked 3589 Read 7 Done New Done true 2591 Acked 3590 Read 7 Done New Done true 2592 Acked 3591 Read 2 Done New Done true 2593 Acked 3592 Flush 0 Done New Done true 2594 Acked 3593 Write 2 Done New Done true 2595 Acked 3594 Write 10 Done New Done true 2596 Acked 3595 Write 9 Done New Done true 2597 Acked 3596 Read 9 Done New Done true 2598 Acked 3597 Read 2 Done New Done true 2599 Acked 3598 Read 10 Done New Done true 2600 Acked 3599 Write 4 Done New Done true 2601 Acked 3600 Read 1 Done New Done true 2602 Acked 3601 Flush 0 Done New Done true 2603 Acked 3602 Read 8 Done New Done true 2604 Acked 3603 Read 6 Done New Done true 2605 Acked 3604 Read 6 Done New Done true 2606 Acked 3605 Read 9 Done New Done true 2607 Acked 3606 Read 5 Done New Done true 2608 Acked 3607 Read 7 Done New Done true 2609 Acked 3608 Flush 0 Done New Done true 2610 Acked 3609 Read 2 Done New Done true 2611 Acked 3610 Read 2 Done New Done true 2612 Acked 3611 Read 8 Done New Done true 2613 Acked 3612 Write 9 Done New Done true 2614 Acked 3613 Flush 0 Done New Done true 2615 Acked 3614 Write 8 Done New Done true 2616 Acked 3615 Write 4 Done New Done true 2617 Acked 3616 Write 2 Done New Done true 2618 Acked 3617 Read 9 Done New Done true 2619 Acked 3618 Write 4 Done New Done true 2620 Acked 3619 Read 5 Done New Done true 2621 Acked 3620 Read 5 Done New Done true 2622 Acked 3621 Read 5 Done New Done true 2623 Acked 3622 Read 3 Done New Done true 2624 Acked 3623 Write 8 Done New Done true 2625 Acked 3624 Write 7 Done New Done true 2626 Acked 3625 Read 4 Done New Done true 2627 Acked 3626 Read 3 Done New Done true 2628 Acked 3627 Write 5 Done New Done true 2629 Acked 3628 Read 2 Done New Done true 2630 Acked 3629 Write 8 Done New Done true 2631 Acked 3630 Read 5 Done New Done true 2632 Acked 3631 Write 3 Done New Done true 2633 Acked 3632 Read 9 Done New Done true 2634 Acked 3633 Write 5 Done New Done true 2635 Acked 3634 Read 7 Done New Done true 2636 Acked 3635 Write 3 Done New Done true 2637 Acked 3636 Write 1 Done New Done true 2638 Acked 3637 Write 10 Done New Done true 2639 Acked 3638 Read 1 Done New Done true 2640 Acked 3639 Read 1 Done New Done true 2641 Acked 3640 Read 7 Done New Done true 2642 Acked 3641 Flush 0 Done New Done true 2643 Acked 3642 Read 2 Done New Done true 2644 Acked 3643 Read 8 Done New Done true 2645 Acked 3644 Read 5 Done New Done true 2646 Acked 3645 Read 3 Done New Done true 2647 Acked 3646 Flush 0 Done New Done true 2648 Acked 3647 Read 3 Done New Done true 2649 Acked 3648 Write 4 Done New Done true 2650 Acked 3649 Read 5 Done New Done true 2651 Acked 3650 Flush 0 Done New Done true 2652 Acked 3651 Write 9 Done New Done true 2653 Acked 3652 Read 3 Done New Done true 2654 Acked 3653 Write 7 Done New Done true 2655 Acked 3654 Read 3 Done New Done true 2656 Acked 3655 Write 4 Done New Done true 2657 Acked 3656 Write 4 Done New Done true 2658 Acked 3657 Read 9 Done New Done true 2659 Acked 3658 Flush 0 Done New Done true 2660 Acked 3659 Read 1 Done New Done true 2661 Acked 3660 Read 2 Done New Done true 2662 Acked 3661 Write 6 Done New Done true 2663 Acked 3662 Write 6 Done New Done true 2664 Acked 3663 Read 9 Done New Done true 2665 Acked 3664 Write 4 Done New Done true 2666 Acked 3665 Read 4 Done New Done true 2667 Acked 3666 Write 10 Done New Done true 2668 Acked 3667 Write 4 Done New Done true 2669 Acked 3668 Write 9 Done New Done true 2670 Acked 3669 Write 10 Done New Done true 2671 Acked 3670 Write 7 Done New Done true 2672 Acked 3671 Write 7 Done New Done true 2673 Acked 3672 Read 7 Done New Done true 2674 Acked 3673 Read 5 Done New Done true 2675 Acked 3674 Write 9 Done New Done true 2676 Acked 3675 Read 7 Done New Done true 2677 Acked 3676 Read 1 Done New Done true 2678 Acked 3677 Read 1 Done New Done true 2679 Acked 3678 Write 5 Done New Done true 2680 Acked 3679 Read 10 Done New Done true 2681 Acked 3680 Read 6 Done New Done true 2682 Acked 3681 Write 3 Done New Done true 2683 Acked 3682 Write 8 Done New Done true 2684 Acked 3683 Read 9 Done New Done true 2685 Acked 3684 Write 4 Done New Done true 2686 Acked 3685 Read 9 Done New Done true 2687 Acked 3686 Write 9 Done New Done true 2688 Acked 3687 Read 8 Done New Done true 2689 Acked 3688 Read 6 Done New Done true 2690 Acked 3689 Read 3 Done New Done true 2691 Acked 3690 Write 7 Done New Done true 2692 Acked 3691 Read 6 Done New Done true 2693 Acked 3692 Write 10 Done New Done true 2694 Acked 3693 Flush 0 Done New Done true 2695 Acked 3694 Write 5 Done New Done true 2696 Acked 3695 Flush 0 Done New Done true 2697 Acked 3696 Read 9 Done New Done true 2698 Acked 3697 Write 5 Done New Done true 2699 Acked 3698 Read 2 Done New Done true 2700 Acked 3699 Read 6 Done New Done true 2701 Acked 3700 Write 6 Done New Done true 2702 Acked 3701 Write 7 Done New Done true 2703 Acked 3702 Write 6 Done New Done true 2704 Acked 3703 Read 10 Done New Done true 2705 Acked 3704 Write 6 Done New Done true 2706 Acked 3705 Read 8 Done New Done true 2707 Acked 3706 Flush 0 Done New Done true 2708 Acked 3707 Read 5 Done New Done true 2709 Acked 3708 Read 6 Done New Done true 2710 Acked 3709 Read 3 Done New Done true 2711 Acked 3710 Write 3 Done New Done true 2712 Acked 3711 Read 4 Done New Done true 2713 Acked 3712 Write 8 Done New Done true 2714 Acked 3713 Write 10 Done New Done true 2715 Acked 3714 Read 6 Done New Done true 2716 Acked 3715 Read 8 Done New Done true 2717 Acked 3716 Write 10 Done New Done true 2718 Acked 3717 Write 5 Done New Done true 2719 Acked 3718 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 301 0 301 Sent 0 0 0 0 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3718 3417 3718 Downstairs last five completed: 3417 3416 3415 3414 3413 Upstairs last five completed: 2719 2718 2717 2716 2715 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:15.990556459Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:15.990624486Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:15.990652484Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:15.990678781Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:15.990699046Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:15.990800839Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:15.991078068Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:16.991158058Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:16.991226656Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:16.991255117Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:16.991279007Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:16.991305008Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:16.991325118Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:16.991348721Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:16.991595809Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:17.992388818Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:17.992455796Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:17.992484125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:17.992510668Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:17.992541271Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:17.992583657Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:17.992823465Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:18.994025301Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:18.994122997Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:18.994151254Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:18.994177465Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:18.994197681Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:18.994302458Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:18.994525212Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2419 Acked 3418 Write 7 Done New Done true 2420 Acked 3419 Read 10 Done New Done true 2421 Acked 3420 Read 3 Done New Done true 2422 Acked 3421 Write 2 Done New Done true 2423 Acked 3422 Write 7 Done New Done true 2424 Acked 3423 Write 5 Done New Done true 2425 Acked 3424 Read 7 Done New Done true 2426 Acked 3425 Flush 0 Done New Done true 2427 Acked 3426 Read 2 Done New Done true 2428 Acked 3427 Write 1 Done New Done true 2429 Acked 3428 Write 4 Done New Done true 2430 Acked 3429 Write 3 Done New Done true 2431 Acked 3430 Read 9 Done New Done true 2432 Acked 3431 Read 5 Done New Done true 2433 Acked 3432 Read 9 Done New Done true 2434 Acked 3433 Write 7 Done New Done true 2435 Acked 3434 Read 1 Done New Done true 2436 Acked 3435 Write 9 Done New Done true 2437 Acked 3436 Read 3 Done New Done true 2438 Acked 3437 Read 3 Done New Done true 2439 Acked 3438 Flush 0 Done New Done true 2440 Acked 3439 Write 1 Done New Done true 2441 Acked 3440 Read 8 Done New Done true 2442 Acked 3441 Write 10 Done New Done true 2443 Acked 3442 Write 9 Done New Done true 2444 Acked 3443 Flush 0 Done New Done true 2445 Acked 3444 Write 3 Done New Done true 2446 Acked 3445 Flush 0 Done New Done true 2447 Acked 3446 Read 10 Done New Done true 2448 Acked 3447 Read 2 Done New Done true 2449 Acked 3448 Write 8 Done New Done true 2450 Acked 3449 Read 7 Done New Done true 2451 Acked 3450 Read 6 Done New Done true 2452 Acked 3451 Write 9 Done New Done true 2453 Acked 3452 Read 8 Done New Done true 2454 Acked 3453 Read 10 Done New Done true 2455 Acked 3454 Read 2 Done New Done true 2456 Acked 3455 Read 7 Done New Done true 2457 Acked 3456 Write 2 Done New Done true 2458 Acked 3457 Write 7 Done New Done true 2459 Acked 3458 Write 1 Done New Done true 2460 Acked 3459 Read 8 Done New Done true 2461 Acked 3460 Write 4 Done New Done true 2462 Acked 3461 Read 5 Done New Done true 2463 Acked 3462 Read 1 Done New Done true 2464 Acked 3463 Read 10 Done New Done true 2465 Acked 3464 Write 1 Done New Done true 2466 Acked 3465 Read 9 Done New Done true 2467 Acked 3466 Write 8 Done New Done true 2468 Acked 3467 Write 4 Done New Done true 2469 Acked 3468 Flush 0 Done New Done true 2470 Acked 3469 Read 2 Done New Done true 2471 Acked 3470 Write 4 Done New Done true 2472 Acked 3471 Write 9 Done New Done true 2473 Acked 3472 Write 2 Done New Done true 2474 Acked 3473 Read 5 Done New Done true 2475 Acked 3474 Flush 0 Done New Done true 2476 Acked 3475 Write 8 Done New Done true 2477 Acked 3476 Read 4 Done New Done true 2478 Acked 3477 Read 6 Done New Done true 2479 Acked 3478 Read 2 Done New Done true 2480 Acked 3479 Write 1 Done New Done true 2481 Acked 3480 Read 10 Done New Done true 2482 Acked 3481 Read 4 Done New Done true 2483 Acked 3482 Read 5 Done New Done true 2484 Acked 3483 Read 4 Done New Done true 2485 Acked 3484 Read 7 Done New Done true 2486 Acked 3485 Read 7 Done New Done true 2487 Acked 3486 Write 9 Done New Done true 2488 Acked 3487 Read 6 Done New Done true 2489 Acked 3488 Write 7 Done New Done true 2490 Acked 3489 Read 9 Done New Done true 2491 Acked 3490 Write 2 Done New Done true 2492 Acked 3491 Read 2 Done New Done true 2493 Acked 3492 Read 6 Done New Done true 2494 Acked 3493 Read 8 Done New Done true 2495 Acked 3494 Write 6 Done New Done true 2496 Acked 3495 Read 3 Done New Done true 2497 Acked 3496 Read 2 Done New Done true 2498 Acked 3497 Read 8 Done New Done true 2499 Acked 3498 Write 6 Done New Done true 2500 Acked 3499 Write 9 Done New Done true 2501 Acked 3500 Read 6 Done New Done true 2502 Acked 3501 Write 5 Done New Done true 2503 Acked 3502 Flush 0 Done New Done true 2504 Acked 3503 Read 1 Done New Done true 2505 Acked 3504 Write 1 Done New Done true 2506 Acked 3505 Read 5 Done New Done true 2507 Acked 3506 Read 8 Done New Done true 2508 Acked 3507 Write 5 Done New Done true 2509 Acked 3508 Write 8 Done New Done true 2510 Acked 3509 Flush 0 Done New Done true 2511 Acked 3510 Flush 0 Done New Done true 2512 Acked 3511 Read 8 Done New Done true 2513 Acked 3512 Write 6 Done New Done true 2514 Acked 3513 Read 6 Done New Done true 2515 Acked 3514 Write 7 Done New Done true 2516 Acked 3515 Read 5 Done New Done true 2517 Acked 3516 Read 7 Done New Done true 2518 Acked 3517 Write 10 Done New Done true 2519 Acked 3518 Read 1 Done New Done true 2520 Acked 3519 Write 5 Done New Done true 2521 Acked 3520 Read 6 Done New Done true 2522 Acked 3521 Read 6 Done New Done true 2523 Acked 3522 Write 2 Done New Done true 2524 Acked 3523 Read 9 Done New Done true 2525 Acked 3524 Read 7 Done New Done true 2526 Acked 3525 Flush 0 Done New Done true 2527 Acked 3526 Read 4 Done New Done true 2528 Acked 3527 Write 3 Done New Done true 2529 Acked 3528 Read 10 Done New Done true 2530 Acked 3529 Read 5 Done New Done true 2531 Acked 3530 Write 1 Done New Done true 2532 Acked 3531 Read 7 Done New Done true 2533 Acked 3532 Write 10 Done New Done true 2534 Acked 3533 Write 6 Done New Done true 2535 Acked 3534 Write 2 Done New Done true 2536 Acked 3535 Read 1 Done New Done true 2537 Acked 3536 Read 3 Done New Done true 2538 Acked 3537 Flush 0 Done New Done true 2539 Acked 3538 Write 1 Done New Done true 2540 Acked 3539 Write 5 Done New Done true 2541 Acked 3540 Write 8 Done New Done true 2542 Acked 3541 Flush 0 Done New Done true 2543 Acked 3542 Write 1 Done New Done true 2544 Acked 3543 Read 4 Done New Done true 2545 Acked 3544 Write 9 Done New Done true 2546 Acked 3545 Write 4 Done New Done true 2547 Acked 3546 Flush 0 Done New Done true 2548 Acked 3547 Read 10 Done New Done true 2549 Acked 3548 Write 7 Done New Done true 2550 Acked 3549 Write 2 Done New Done true 2551 Acked 3550 Flush 0 Done New Done true 2552 Acked 3551 Write 8 Done New Done true 2553 Acked 3552 Write 5 Done New Done true 2554 Acked 3553 Write 6 Done New Done true 2555 Acked 3554 Write 2 Done New Done true 2556 Acked 3555 Flush 0 Done New Done true 2557 Acked 3556 Read 9 Done New Done true 2558 Acked 3557 Read 7 Done New Done true 2559 Acked 3558 Read 1 Done New Done true 2560 Acked 3559 Read 10 Done New Done true 2561 Acked 3560 Flush 0 Done New Done true 2562 Acked 3561 Read 7 Done New Done true 2563 Acked 3562 Read 1 Done New Done true 2564 Acked 3563 Write 3 Done New Done true 2565 Acked 3564 Flush 0 Done New Done true 2566 Acked 3565 Write 1 Done New Done true 2567 Acked 3566 Write 1 Done New Done true 2568 Acked 3567 Read 7 Done New Done true 2569 Acked 3568 Read 3 Done New Done true 2570 Acked 3569 Read 4 Done New Done true 2571 Acked 3570 Write 3 Done New Done true 2572 Acked 3571 Read 3 Done New Done true 2573 Acked 3572 Read 8 Done New Done true 2574 Acked 3573 Flush 0 Done New Done true 2575 Acked 3574 Write 4 Done New Done true 2576 Acked 3575 Read 5 Done New Done true 2577 Acked 3576 Flush 0 Done New Done true 2578 Acked 3577 Flush 0 Done New Done true 2579 Acked 3578 Read 9 Done New Done true 2580 Acked 3579 Write 9 Done New Done true 2581 Acked 3580 Write 3 Done New Done true 2582 Acked 3581 Write 7 Done New Done true 2583 Acked 3582 Read 5 Done New Done true 2584 Acked 3583 Write 6 Done New Done true 2585 Acked 3584 Read 8 Done New Done true 2586 Acked 3585 Flush 0 Done New Done true 2587 Acked 3586 Read 9 Done New Done true 2588 Acked 3587 Read 2 Done New Done true 2589 Acked 3588 Write 9 Done New Done true 2590 Acked 3589 Read 7 Done New Done true 2591 Acked 3590 Read 7 Done New Done true 2592 Acked 3591 Read 2 Done New Done true 2593 Acked 3592 Flush 0 Done New Done true 2594 Acked 3593 Write 2 Done New Done true 2595 Acked 3594 Write 10 Done New Done true 2596 Acked 3595 Write 9 Done New Done true 2597 Acked 3596 Read 9 Done New Done true 2598 Acked 3597 Read 2 Done New Done true 2599 Acked 3598 Read 10 Done New Done true 2600 Acked 3599 Write 4 Done New Done true 2601 Acked 3600 Read 1 Done New Done true 2602 Acked 3601 Flush 0 Done New Done true 2603 Acked 3602 Read 8 Done New Done true 2604 Acked 3603 Read 6 Done New Done true 2605 Acked 3604 Read 6 Done New Done true 2606 Acked 3605 Read 9 Done New Done true 2607 Acked 3606 Read 5 Done New Done true 2608 Acked 3607 Read 7 Done New Done true 2609 Acked 3608 Flush 0 Done New Done true 2610 Acked 3609 Read 2 Done New Done true 2611 Acked 3610 Read 2 Done New Done true 2612 Acked 3611 Read 8 Done New Done true 2613 Acked 3612 Write 9 Done New Done true 2614 Acked 3613 Flush 0 Done New Done true 2615 Acked 3614 Write 8 Done New Done true 2616 Acked 3615 Write 4 Done New Done true 2617 Acked 3616 Write 2 Done New Done true 2618 Acked 3617 Read 9 Done New Done true 2619 Acked 3618 Write 4 Done New Done true 2620 Acked 3619 Read 5 Done New Done true 2621 Acked 3620 Read 5 Done New Done true 2622 Acked 3621 Read 5 Done New Done true 2623 Acked 3622 Read 3 Done New Done true 2624 Acked 3623 Write 8 Done New Done true 2625 Acked 3624 Write 7 Done New Done true 2626 Acked 3625 Read 4 Done New Done true 2627 Acked 3626 Read 3 Done New Done true 2628 Acked 3627 Write 5 Done New Done true 2629 Acked 3628 Read 2 Done New Done true 2630 Acked 3629 Write 8 Done New Done true 2631 Acked 3630 Read 5 Done New Done true 2632 Acked 3631 Write 3 Done New Done true 2633 Acked 3632 Read 9 Done New Done true 2634 Acked 3633 Write 5 Done New Done true 2635 Acked 3634 Read 7 Done New Done true 2636 Acked 3635 Write 3 Done New Done true 2637 Acked 3636 Write 1 Done New Done true 2638 Acked 3637 Write 10 Done New Done true 2639 Acked 3638 Read 1 Done New Done true 2640 Acked 3639 Read 1 Done New Done true 2641 Acked 3640 Read 7 Done New Done true 2642 Acked 3641 Flush 0 Done New Done true 2643 Acked 3642 Read 2 Done New Done true 2644 Acked 3643 Read 8 Done New Done true 2645 Acked 3644 Read 5 Done New Done true 2646 Acked 3645 Read 3 Done New Done true 2647 Acked 3646 Flush 0 Done New Done true 2648 Acked 3647 Read 3 Done New Done true 2649 Acked 3648 Write 4 Done New Done true 2650 Acked 3649 Read 5 Done New Done true 2651 Acked 3650 Flush 0 Done New Done true 2652 Acked 3651 Write 9 Done New Done true 2653 Acked 3652 Read 3 Done New Done true 2654 Acked 3653 Write 7 Done New Done true 2655 Acked 3654 Read 3 Done New Done true 2656 Acked 3655 Write 4 Done New Done true 2657 Acked 3656 Write 4 Done New Done true 2658 Acked 3657 Read 9 Done New Done true 2659 Acked 3658 Flush 0 Done New Done true 2660 Acked 3659 Read 1 Done New Done true 2661 Acked 3660 Read 2 Done New Done true 2662 Acked 3661 Write 6 Done New Done true 2663 Acked 3662 Write 6 Done New Done true 2664 Acked 3663 Read 9 Done New Done true 2665 Acked 3664 Write 4 Done New Done true 2666 Acked 3665 Read 4 Done New Done true 2667 Acked 3666 Write 10 Done New Done true 2668 Acked 3667 Write 4 Done New Done true 2669 Acked 3668 Write 9 Done New Done true 2670 Acked 3669 Write 10 Done New Done true 2671 Acked 3670 Write 7 Done New Done true 2672 Acked 3671 Write 7 Done New Done true 2673 Acked 3672 Read 7 Done New Done true 2674 Acked 3673 Read 5 Done New Done true 2675 Acked 3674 Write 9 Done New Done true 2676 Acked 3675 Read 7 Done New Done true 2677 Acked 3676 Read 1 Done New Done true 2678 Acked 3677 Read 1 Done New Done true 2679 Acked 3678 Write 5 Done New Done true 2680 Acked 3679 Read 10 Done New Done true 2681 Acked 3680 Read 6 Done New Done true 2682 Acked 3681 Write 3 Done New Done true 2683 Acked 3682 Write 8 Done New Done true 2684 Acked 3683 Read 9 Done New Done true 2685 Acked 3684 Write 4 Done New Done true 2686 Acked 3685 Read 9 Done New Done true 2687 Acked 3686 Write 9 Done New Done true 2688 Acked 3687 Read 8 Done New Done true 2689 Acked 3688 Read 6 Done New Done true 2690 Acked 3689 Read 3 Done New Done true 2691 Acked 3690 Write 7 Done New Done true 2692 Acked 3691 Read 6 Done New Done true 2693 Acked 3692 Write 10 Done New Done true 2694 Acked 3693 Flush 0 Done New Done true 2695 Acked 3694 Write 5 Done New Done true 2696 Acked 3695 Flush 0 Done New Done true 2697 Acked 3696 Read 9 Done New Done true 2698 Acked 3697 Write 5 Done New Done true 2699 Acked 3698 Read 2 Done New Done true 2700 Acked 3699 Read 6 Done New Done true 2701 Acked 3700 Write 6 Done New Done true 2702 Acked 3701 Write 7 Done New Done true 2703 Acked 3702 Write 6 Done New Done true 2704 Acked 3703 Read 10 Done New Done true 2705 Acked 3704 Write 6 Done New Done true 2706 Acked 3705 Read 8 Done New Done true 2707 Acked 3706 Flush 0 Done New Done true 2708 Acked 3707 Read 5 Done New Done true 2709 Acked 3708 Read 6 Done New Done true 2710 Acked 3709 Read 3 Done New Done true 2711 Acked 3710 Write 3 Done New Done true 2712 Acked 3711 Read 4 Done New Done true 2713 Acked 3712 Write 8 Done New Done true 2714 Acked 3713 Write 10 Done New Done true 2715 Acked 3714 Read 6 Done New Done true 2716 Acked 3715 Read 8 Done New Done true 2717 Acked 3716 Write 10 Done New Done true 2718 Acked 3717 Write 5 Done New Done true 2719 Acked 3718 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 301 0 301 Sent 0 0 0 0 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3718 3417 3718 Downstairs last five completed: 3417 3416 3415 3414 3413 Upstairs last five completed: 2719 2718 2717 2716 2715 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:19.995781549Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:19.995895151Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:19.995923468Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:19.995949851Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:19.995970093Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:19.996065712Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:19.996309753Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:20.997487852Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:20.997565381Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:20.997594558Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:20.997621072Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:20.997641265Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:20.997682035Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:20.997959491Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:21.998629681Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:21.998695515Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:21.998723683Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:21.998748269Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:21.998807021Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:21.99882934Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:21.998869474Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:21.999117868Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:22.999971606Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:23.000038271Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:23.000066685Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:23.000093315Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:23.000113711Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:23.000156211Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:23.000389692Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2419 Acked 3418 Write 7 Done New Done true 2420 Acked 3419 Read 10 Done New Done true 2421 Acked 3420 Read 3 Done New Done true 2422 Acked 3421 Write 2 Done New Done true 2423 Acked 3422 Write 7 Done New Done true 2424 Acked 3423 Write 5 Done New Done true 2425 Acked 3424 Read 7 Done New Done true 2426 Acked 3425 Flush 0 Done New Done true 2427 Acked 3426 Read 2 Done New Done true 2428 Acked 3427 Write 1 Done New Done true 2429 Acked 3428 Write 4 Done New Done true 2430 Acked 3429 Write 3 Done New Done true 2431 Acked 3430 Read 9 Done New Done true 2432 Acked 3431 Read 5 Done New Done true 2433 Acked 3432 Read 9 Done New Done true 2434 Acked 3433 Write 7 Done New Done true 2435 Acked 3434 Read 1 Done New Done true 2436 Acked 3435 Write 9 Done New Done true 2437 Acked 3436 Read 3 Done New Done true 2438 Acked 3437 Read 3 Done New Done true 2439 Acked 3438 Flush 0 Done New Done true 2440 Acked 3439 Write 1 Done New Done true 2441 Acked 3440 Read 8 Done New Done true 2442 Acked 3441 Write 10 Done New Done true 2443 Acked 3442 Write 9 Done New Done true 2444 Acked 3443 Flush 0 Done New Done true 2445 Acked 3444 Write 3 Done New Done true 2446 Acked 3445 Flush 0 Done New Done true 2447 Acked 3446 Read 10 Done New Done true 2448 Acked 3447 Read 2 Done New Done true 2449 Acked 3448 Write 8 Done New Done true 2450 Acked 3449 Read 7 Done New Done true 2451 Acked 3450 Read 6 Done New Done true 2452 Acked 3451 Write 9 Done New Done true 2453 Acked 3452 Read 8 Done New Done true 2454 Acked 3453 Read 10 Done New Done true 2455 Acked 3454 Read 2 Done New Done true 2456 Acked 3455 Read 7 Done New Done true 2457 Acked 3456 Write 2 Done New Done true 2458 Acked 3457 Write 7 Done New Done true 2459 Acked 3458 Write 1 Done New Done true 2460 Acked 3459 Read 8 Done New Done true 2461 Acked 3460 Write 4 Done New Done true 2462 Acked 3461 Read 5 Done New Done true 2463 Acked 3462 Read 1 Done New Done true 2464 Acked 3463 Read 10 Done New Done true 2465 Acked 3464 Write 1 Done New Done true 2466 Acked 3465 Read 9 Done New Done true 2467 Acked 3466 Write 8 Done New Done true 2468 Acked 3467 Write 4 Done New Done true 2469 Acked 3468 Flush 0 Done New Done true 2470 Acked 3469 Read 2 Done New Done true 2471 Acked 3470 Write 4 Done New Done true 2472 Acked 3471 Write 9 Done New Done true 2473 Acked 3472 Write 2 Done New Done true 2474 Acked 3473 Read 5 Done New Done true 2475 Acked 3474 Flush 0 Done New Done true 2476 Acked 3475 Write 8 Done New Done true 2477 Acked 3476 Read 4 Done New Done true 2478 Acked 3477 Read 6 Done New Done true 2479 Acked 3478 Read 2 Done New Done true 2480 Acked 3479 Write 1 Done New Done true 2481 Acked 3480 Read 10 Done New Done true 2482 Acked 3481 Read 4 Done New Done true 2483 Acked 3482 Read 5 Done New Done true 2484 Acked 3483 Read 4 Done New Done true 2485 Acked 3484 Read 7 Done New Done true 2486 Acked 3485 Read 7 Done New Done true 2487 Acked 3486 Write 9 Done New Done true 2488 Acked 3487 Read 6 Done New Done true 2489 Acked 3488 Write 7 Done New Done true 2490 Acked 3489 Read 9 Done New Done true 2491 Acked 3490 Write 2 Done New Done true 2492 Acked 3491 Read 2 Done New Done true 2493 Acked 3492 Read 6 Done New Done true 2494 Acked 3493 Read 8 Done New Done true 2495 Acked 3494 Write 6 Done New Done true 2496 Acked 3495 Read 3 Done New Done true 2497 Acked 3496 Read 2 Done New Done true 2498 Acked 3497 Read 8 Done New Done true 2499 Acked 3498 Write 6 Done New Done true 2500 Acked 3499 Write 9 Done New Done true 2501 Acked 3500 Read 6 Done New Done true 2502 Acked 3501 Write 5 Done New Done true 2503 Acked 3502 Flush 0 Done New Done true 2504 Acked 3503 Read 1 Done New Done true 2505 Acked 3504 Write 1 Done New Done true 2506 Acked 3505 Read 5 Done New Done true 2507 Acked 3506 Read 8 Done New Done true 2508 Acked 3507 Write 5 Done New Done true 2509 Acked 3508 Write 8 Done New Done true 2510 Acked 3509 Flush 0 Done New Done true 2511 Acked 3510 Flush 0 Done New Done true 2512 Acked 3511 Read 8 Done New Done true 2513 Acked 3512 Write 6 Done New Done true 2514 Acked 3513 Read 6 Done New Done true 2515 Acked 3514 Write 7 Done New Done true 2516 Acked 3515 Read 5 Done New Done true 2517 Acked 3516 Read 7 Done New Done true 2518 Acked 3517 Write 10 Done New Done true 2519 Acked 3518 Read 1 Done New Done true 2520 Acked 3519 Write 5 Done New Done true 2521 Acked 3520 Read 6 Done New Done true 2522 Acked 3521 Read 6 Done New Done true 2523 Acked 3522 Write 2 Done New Done true 2524 Acked 3523 Read 9 Done New Done true 2525 Acked 3524 Read 7 Done New Done true 2526 Acked 3525 Flush 0 Done New Done true 2527 Acked 3526 Read 4 Done New Done true 2528 Acked 3527 Write 3 Done New Done true 2529 Acked 3528 Read 10 Done New Done true 2530 Acked 3529 Read 5 Done New Done true 2531 Acked 3530 Write 1 Done New Done true 2532 Acked 3531 Read 7 Done New Done true 2533 Acked 3532 Write 10 Done New Done true 2534 Acked 3533 Write 6 Done New Done true 2535 Acked 3534 Write 2 Done New Done true 2536 Acked 3535 Read 1 Done New Done true 2537 Acked 3536 Read 3 Done New Done true 2538 Acked 3537 Flush 0 Done New Done true 2539 Acked 3538 Write 1 Done New Done true 2540 Acked 3539 Write 5 Done New Done true 2541 Acked 3540 Write 8 Done New Done true 2542 Acked 3541 Flush 0 Done New Done true 2543 Acked 3542 Write 1 Done New Done true 2544 Acked 3543 Read 4 Done New Done true 2545 Acked 3544 Write 9 Done New Done true 2546 Acked 3545 Write 4 Done New Done true 2547 Acked 3546 Flush 0 Done New Done true 2548 Acked 3547 Read 10 Done New Done true 2549 Acked 3548 Write 7 Done New Done true 2550 Acked 3549 Write 2 Done New Done true 2551 Acked 3550 Flush 0 Done New Done true 2552 Acked 3551 Write 8 Done New Done true 2553 Acked 3552 Write 5 Done New Done true 2554 Acked 3553 Write 6 Done New Done true 2555 Acked 3554 Write 2 Done New Done true 2556 Acked 3555 Flush 0 Done New Done true 2557 Acked 3556 Read 9 Done New Done true 2558 Acked 3557 Read 7 Done New Done true 2559 Acked 3558 Read 1 Done New Done true 2560 Acked 3559 Read 10 Done New Done true 2561 Acked 3560 Flush 0 Done New Done true 2562 Acked 3561 Read 7 Done New Done true 2563 Acked 3562 Read 1 Done New Done true 2564 Acked 3563 Write 3 Done New Done true 2565 Acked 3564 Flush 0 Done New Done true 2566 Acked 3565 Write 1 Done New Done true 2567 Acked 3566 Write 1 Done New Done true 2568 Acked 3567 Read 7 Done New Done true 2569 Acked 3568 Read 3 Done New Done true 2570 Acked 3569 Read 4 Done New Done true 2571 Acked 3570 Write 3 Done New Done true 2572 Acked 3571 Read 3 Done New Done true 2573 Acked 3572 Read 8 Done New Done true 2574 Acked 3573 Flush 0 Done New Done true 2575 Acked 3574 Write 4 Done New Done true 2576 Acked 3575 Read 5 Done New Done true 2577 Acked 3576 Flush 0 Done New Done true 2578 Acked 3577 Flush 0 Done New Done true 2579 Acked 3578 Read 9 Done New Done true 2580 Acked 3579 Write 9 Done New Done true 2581 Acked 3580 Write 3 Done New Done true 2582 Acked 3581 Write 7 Done New Done true 2583 Acked 3582 Read 5 Done New Done true 2584 Acked 3583 Write 6 Done New Done true 2585 Acked 3584 Read 8 Done New Done true 2586 Acked 3585 Flush 0 Done New Done true 2587 Acked 3586 Read 9 Done New Done true 2588 Acked 3587 Read 2 Done New Done true 2589 Acked 3588 Write 9 Done New Done true 2590 Acked 3589 Read 7 Done New Done true 2591 Acked 3590 Read 7 Done New Done true 2592 Acked 3591 Read 2 Done New Done true 2593 Acked 3592 Flush 0 Done New Done true 2594 Acked 3593 Write 2 Done New Done true 2595 Acked 3594 Write 10 Done New Done true 2596 Acked 3595 Write 9 Done New Done true 2597 Acked 3596 Read 9 Done New Done true 2598 Acked 3597 Read 2 Done New Done true 2599 Acked 3598 Read 10 Done New Done true 2600 Acked 3599 Write 4 Done New Done true 2601 Acked 3600 Read 1 Done New Done true 2602 Acked 3601 Flush 0 Done New Done true 2603 Acked 3602 Read 8 Done New Done true 2604 Acked 3603 Read 6 Done New Done true 2605 Acked 3604 Read 6 Done New Done true 2606 Acked 3605 Read 9 Done New Done true 2607 Acked 3606 Read 5 Done New Done true 2608 Acked 3607 Read 7 Done New Done true 2609 Acked 3608 Flush 0 Done New Done true 2610 Acked 3609 Read 2 Done New Done true 2611 Acked 3610 Read 2 Done New Done true 2612 Acked 3611 Read 8 Done New Done true 2613 Acked 3612 Write 9 Done New Done true 2614 Acked 3613 Flush 0 Done New Done true 2615 Acked 3614 Write 8 Done New Done true 2616 Acked 3615 Write 4 Done New Done true 2617 Acked 3616 Write 2 Done New Done true 2618 Acked 3617 Read 9 Done New Done true 2619 Acked 3618 Write 4 Done New Done true 2620 Acked 3619 Read 5 Done New Done true 2621 Acked 3620 Read 5 Done New Done true 2622 Acked 3621 Read 5 Done New Done true 2623 Acked 3622 Read 3 Done New Done true 2624 Acked 3623 Write 8 Done New Done true 2625 Acked 3624 Write 7 Done New Done true 2626 Acked 3625 Read 4 Done New Done true 2627 Acked 3626 Read 3 Done New Done true 2628 Acked 3627 Write 5 Done New Done true 2629 Acked 3628 Read 2 Done New Done true 2630 Acked 3629 Write 8 Done New Done true 2631 Acked 3630 Read 5 Done New Done true 2632 Acked 3631 Write 3 Done New Done true 2633 Acked 3632 Read 9 Done New Done true 2634 Acked 3633 Write 5 Done New Done true 2635 Acked 3634 Read 7 Done New Done true 2636 Acked 3635 Write 3 Done New Done true 2637 Acked 3636 Write 1 Done New Done true 2638 Acked 3637 Write 10 Done New Done true 2639 Acked 3638 Read 1 Done New Done true 2640 Acked 3639 Read 1 Done New Done true 2641 Acked 3640 Read 7 Done New Done true 2642 Acked 3641 Flush 0 Done New Done true 2643 Acked 3642 Read 2 Done New Done true 2644 Acked 3643 Read 8 Done New Done true 2645 Acked 3644 Read 5 Done New Done true 2646 Acked 3645 Read 3 Done New Done true 2647 Acked 3646 Flush 0 Done New Done true 2648 Acked 3647 Read 3 Done New Done true 2649 Acked 3648 Write 4 Done New Done true 2650 Acked 3649 Read 5 Done New Done true 2651 Acked 3650 Flush 0 Done New Done true 2652 Acked 3651 Write 9 Done New Done true 2653 Acked 3652 Read 3 Done New Done true 2654 Acked 3653 Write 7 Done New Done true 2655 Acked 3654 Read 3 Done New Done true 2656 Acked 3655 Write 4 Done New Done true 2657 Acked 3656 Write 4 Done New Done true 2658 Acked 3657 Read 9 Done New Done true 2659 Acked 3658 Flush 0 Done New Done true 2660 Acked 3659 Read 1 Done New Done true 2661 Acked 3660 Read 2 Done New Done true 2662 Acked 3661 Write 6 Done New Done true 2663 Acked 3662 Write 6 Done New Done true 2664 Acked 3663 Read 9 Done New Done true 2665 Acked 3664 Write 4 Done New Done true 2666 Acked 3665 Read 4 Done New Done true 2667 Acked 3666 Write 10 Done New Done true 2668 Acked 3667 Write 4 Done New Done true 2669 Acked 3668 Write 9 Done New Done true 2670 Acked 3669 Write 10 Done New Done true 2671 Acked 3670 Write 7 Done New Done true 2672 Acked 3671 Write 7 Done New Done true 2673 Acked 3672 Read 7 Done New Done true 2674 Acked 3673 Read 5 Done New Done true 2675 Acked 3674 Write 9 Done New Done true 2676 Acked 3675 Read 7 Done New Done true 2677 Acked 3676 Read 1 Done New Done true 2678 Acked 3677 Read 1 Done New Done true 2679 Acked 3678 Write 5 Done New Done true 2680 Acked 3679 Read 10 Done New Done true 2681 Acked 3680 Read 6 Done New Done true 2682 Acked 3681 Write 3 Done New Done true 2683 Acked 3682 Write 8 Done New Done true 2684 Acked 3683 Read 9 Done New Done true 2685 Acked 3684 Write 4 Done New Done true 2686 Acked 3685 Read 9 Done New Done true 2687 Acked 3686 Write 9 Done New Done true 2688 Acked 3687 Read 8 Done New Done true 2689 Acked 3688 Read 6 Done New Done true 2690 Acked 3689 Read 3 Done New Done true 2691 Acked 3690 Write 7 Done New Done true 2692 Acked 3691 Read 6 Done New Done true 2693 Acked 3692 Write 10 Done New Done true 2694 Acked 3693 Flush 0 Done New Done true 2695 Acked 3694 Write 5 Done New Done true 2696 Acked 3695 Flush 0 Done New Done true 2697 Acked 3696 Read 9 Done New Done true 2698 Acked 3697 Write 5 Done New Done true 2699 Acked 3698 Read 2 Done New Done true 2700 Acked 3699 Read 6 Done New Done true 2701 Acked 3700 Write 6 Done New Done true 2702 Acked 3701 Write 7 Done New Done true 2703 Acked 3702 Write 6 Done New Done true 2704 Acked 3703 Read 10 Done New Done true 2705 Acked 3704 Write 6 Done New Done true 2706 Acked 3705 Read 8 Done New Done true 2707 Acked 3706 Flush 0 Done New Done true 2708 Acked 3707 Read 5 Done New Done true 2709 Acked 3708 Read 6 Done New Done true 2710 Acked 3709 Read 3 Done New Done true 2711 Acked 3710 Write 3 Done New Done true 2712 Acked 3711 Read 4 Done New Done true 2713 Acked 3712 Write 8 Done New Done true 2714 Acked 3713 Write 10 Done New Done true 2715 Acked 3714 Read 6 Done New Done true 2716 Acked 3715 Read 8 Done New Done true 2717 Acked 3716 Write 10 Done New Done true 2718 Acked 3717 Write 5 Done New Done true 2719 Acked 3718 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 301 0 301 Sent 0 0 0 0 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3718 3417 3718 Downstairs last five completed: 3417 3416 3415 3414 3413 Upstairs last five completed: 2719 2718 2717 2716 2715 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:24.001384425Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:24.001452389Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:24.001480248Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:24.001506801Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:24.00152713Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:24.001626916Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:24.001867982Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:25.002654607Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:25.002722634Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:25.002760814Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:25.002788295Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:25.002808809Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:25.002852856Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:25.00309385Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1194) [8820][1] reports Running {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:26.0039951Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:26.004064141Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:26.004093402Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:26.004120319Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:26.004140973Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:26.004182561Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:26.004481136Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.005642413Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.00582193Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.00777576Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.007822016Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.007847338Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 3417","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.008128387Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.008162398Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.00818605Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.008208496Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:27.008240288Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:244 States: Active Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2476 Acked 3475 Write 8 Done Done Done true 2477 Acked 3476 Read 4 Done Done Done true 2478 Acked 3477 Read 6 Done Done Done true 2479 Acked 3478 Read 2 Done Done Done true 2480 Acked 3479 Write 1 Done Done Done true 2481 Acked 3480 Read 10 Done Done Done true 2482 Acked 3481 Read 4 Done Done Done true 2483 Acked 3482 Read 5 Done Done Done true 2484 Acked 3483 Read 4 Done Done Done true 2485 Acked 3484 Read 7 Done Done Done true 2486 Acked 3485 Read 7 Done Done Done true 2487 Acked 3486 Write 9 Done Done Done true 2488 Acked 3487 Read 6 Done Done Done true 2489 Acked 3488 Write 7 Done Done Done true 2490 Acked 3489 Read 9 Done Done Done true 2491 Acked 3490 Write 2 Done Done Done true 2492 Acked 3491 Read 2 Done Done Done true 2493 Acked 3492 Read 6 Done Done Done true 2494 Acked 3493 Read 8 Done Done Done true 2495 Acked 3494 Write 6 Done Done Done true 2496 Acked 3495 Read 3 Done Done Done true 2497 Acked 3496 Read 2 Done Done Done true 2498 Acked 3497 Read 8 Done Done Done true 2499 Acked 3498 Write 6 Done Done Done true 2500 Acked 3499 Write 9 Done Done Done true 2501 Acked 3500 Read 6 Done Done Done true 2502 Acked 3501 Write 5 Done Sent Done true 2503 Acked 3502 Flush 0 Done Sent Done true 2504 Acked 3503 Read 1 Done Sent Done true 2505 Acked 3504 Write 1 Done Sent Done true 2506 Acked 3505 Read 5 Done Sent Done true 2507 Acked 3506 Read 8 Done Sent Done true 2508 Acked 3507 Write 5 Done Sent Done true 2509 Acked 3508 Write 8 Done Sent Done true 2510 Acked 3509 Flush 0 Done Sent Done true 2511 Acked 3510 Flush 0 Done Sent Done true 2512 Acked 3511 Read 8 Done Sent Done true 2513 Acked 3512 Write 6 Done Sent Done true 2514 Acked 3513 Read 6 Done Sent Done true 2515 Acked 3514 Write 7 Done Sent Done true 2516 Acked 3515 Read 5 Done Sent Done true 2517 Acked 3516 Read 7 Done Sent Done true 2518 Acked 3517 Write 10 Done Sent Done true 2519 Acked 3518 Read 1 Done Sent Done true 2520 Acked 3519 Write 5 Done Sent Done true 2521 Acked 3520 Read 6 Done Sent Done true 2522 Acked 3521 Read 6 Done Sent Done true 2523 Acked 3522 Write 2 Done Sent Done true 2524 Acked 3523 Read 9 Done Sent Done true 2525 Acked 3524 Read 7 Done Sent Done true 2526 Acked 3525 Flush 0 Done Sent Done true 2527 Acked 3526 Read 4 Done Sent Done true 2528 Acked 3527 Write 3 Done Sent Done true 2529 Acked 3528 Read 10 Done Sent Done true 2530 Acked 3529 Read 5 Done Sent Done true 2531 Acked 3530 Write 1 Done Sent Done true 2532 Acked 3531 Read 7 Done Sent Done true 2533 Acked 3532 Write 10 Done Sent Done true 2534 Acked 3533 Write 6 Done Sent Done true 2535 Acked 3534 Write 2 Done Sent Done true 2536 Acked 3535 Read 1 Done Sent Done true 2537 Acked 3536 Read 3 Done Sent Done true 2538 Acked 3537 Flush 0 Done Sent Done true 2539 Acked 3538 Write 1 Done Sent Done true 2540 Acked 3539 Write 5 Done Sent Done true 2541 Acked 3540 Write 8 Done Sent Done true 2542 Acked 3541 Flush 0 Done Sent Done true 2543 Acked 3542 Write 1 Done Sent Done true 2544 Acked 3543 Read 4 Done Sent Done true 2545 Acked 3544 Write 9 Done Sent Done true 2546 Acked 3545 Write 4 Done Sent Done true 2547 Acked 3546 Flush 0 Done Sent Done true 2548 Acked 3547 Read 10 Done Sent Done true 2549 Acked 3548 Write 7 Done Sent Done true 2550 Acked 3549 Write 2 Done Sent Done true 2551 Acked 3550 Flush 0 Done Sent Done true 2552 Acked 3551 Write 8 Done Sent Done true 2553 Acked 3552 Write 5 Done Sent Done true 2554 Acked 3553 Write 6 Done Sent Done true 2555 Acked 3554 Write 2 Done Sent Done true 2556 Acked 3555 Flush 0 Done Sent Done true 2557 Acked 3556 Read 9 Done Sent Done true 2558 Acked 3557 Read 7 Done Sent Done true 2559 Acked 3558 Read 1 Done Sent Done true 2560 Acked 3559 Read 10 Done Sent Done true 2561 Acked 3560 Flush 0 Done Sent Done true 2562 Acked 3561 Read 7 Done Sent Done true 2563 Acked 3562 Read 1 Done Sent Done true 2564 Acked 3563 Write 3 Done Sent Done true 2565 Acked 3564 Flush 0 Done Sent Done true 2566 Acked 3565 Write 1 Done Sent Done true 2567 Acked 3566 Write 1 Done Sent Done true 2568 Acked 3567 Read 7 Done Sent Done true 2569 Acked 3568 Read 3 Done Sent Done true 2570 Acked 3569 Read 4 Done Sent Done true 2571 Acked 3570 Write 3 Done Sent Done true 2572 Acked 3571 Read 3 Done Sent Done true 2573 Acked 3572 Read 8 Done Sent Done true 2574 Acked 3573 Flush 0 Done Sent Done true 2575 Acked 3574 Write 4 Done Sent Done true 2576 Acked 3575 Read 5 Done Sent Done true 2577 Acked 3576 Flush 0 Done Sent Done true 2578 Acked 3577 Flush 0 Done Sent Done true 2579 Acked 3578 Read 9 Done Sent Done true 2580 Acked 3579 Write 9 Done Sent Done true 2581 Acked 3580 Write 3 Done Sent Done true 2582 Acked 3581 Write 7 Done Sent Done true 2583 Acked 3582 Read 5 Done Sent Done true 2584 Acked 3583 Write 6 Done Sent Done true 2585 Acked 3584 Read 8 Done Sent Done true 2586 Acked 3585 Flush 0 Done Sent Done true 2587 Acked 3586 Read 9 Done Sent Done true 2588 Acked 3587 Read 2 Done Sent Done true 2589 Acked 3588 Write 9 Done Sent Done true 2590 Acked 3589 Read 7 Done Sent Done true 2591 Acked 3590 Read 7 Done Sent Done true 2592 Acked 3591 Read 2 Done Sent Done true 2593 Acked 3592 Flush 0 Done Sent Done true 2594 Acked 3593 Write 2 Done Sent Done true 2595 Acked 3594 Write 10 Done Sent Done true 2596 Acked 3595 Write 9 Done Sent Done true 2597 Acked 3596 Read 9 Done Sent Done true 2598 Acked 3597 Read 2 Done Sent Done true 2599 Acked 3598 Read 10 Done Sent Done true 2600 Acked 3599 Write 4 Done Sent Done true 2601 Acked 3600 Read 1 Done Sent Done true 2602 Acked 3601 Flush 0 Done Sent Done true 2603 Acked 3602 Read 8 Done Sent Done true 2604 Acked 3603 Read 6 Done Sent Done true 2605 Acked 3604 Read 6 Done Sent Done true 2606 Acked 3605 Read 9 Done Sent Done true 2607 Acked 3606 Read 5 Done Sent Done true 2608 Acked 3607 Read 7 Done Sent Done true 2609 Acked 3608 Flush 0 Done Sent Done true 2610 Acked 3609 Read 2 Done Sent Done true 2611 Acked 3610 Read 2 Done Sent Done true 2612 Acked 3611 Read 8 Done Sent Done true 2613 Acked 3612 Write 9 Done Sent Done true 2614 Acked 3613 Flush 0 Done Sent Done true 2615 Acked 3614 Write 8 Done Sent Done true 2616 Acked 3615 Write 4 Done Sent Done true 2617 Acked 3616 Write 2 Done Sent Done true 2618 Acked 3617 Read 9 Done Sent Done true 2619 Acked 3618 Write 4 Done Sent Done true 2620 Acked 3619 Read 5 Done Sent Done true 2621 Acked 3620 Read 5 Done Sent Done true 2622 Acked 3621 Read 5 Done Sent Done true 2623 Acked 3622 Read 3 Done Sent Done true 2624 Acked 3623 Write 8 Done Sent Done true 2625 Acked 3624 Write 7 Done Sent Done true 2626 Acked 3625 Read 4 Done Sent Done true 2627 Acked 3626 Read 3 Done Sent Done true 2628 Acked 3627 Write 5 Done Sent Done true 2629 Acked 3628 Read 2 Done Sent Done true 2630 Acked 3629 Write 8 Done Sent Done true 2631 Acked 3630 Read 5 Done Sent Done true 2632 Acked 3631 Write 3 Done Sent Done true 2633 Acked 3632 Read 9 Done Sent Done true 2634 Acked 3633 Write 5 Done Sent Done true 2635 Acked 3634 Read 7 Done Sent Done true 2636 Acked 3635 Write 3 Done Sent Done true 2637 Acked 3636 Write 1 Done Sent Done true 2638 Acked 3637 Write 10 Done Sent Done true 2639 Acked 3638 Read 1 Done Sent Done true 2640 Acked 3639 Read 1 Done Sent Done true 2641 Acked 3640 Read 7 Done Sent Done true 2642 Acked 3641 Flush 0 Done Sent Done true 2643 Acked 3642 Read 2 Done Sent Done true 2644 Acked 3643 Read 8 Done Sent Done true 2645 Acked 3644 Read 5 Done Sent Done true 2646 Acked 3645 Read 3 Done Sent Done true 2647 Acked 3646 Flush 0 Done Sent Done true 2648 Acked 3647 Read 3 Done Sent Done true 2649 Acked 3648 Write 4 Done Sent Done true 2650 Acked 3649 Read 5 Done Sent Done true 2651 Acked 3650 Flush 0 Done Sent Done true 2652 Acked 3651 Write 9 Done Sent Done true 2653 Acked 3652 Read 3 Done Sent Done true 2654 Acked 3653 Write 7 Done Sent Done true 2655 Acked 3654 Read 3 Done Sent Done true 2656 Acked 3655 Write 4 Done Sent Done true 2657 Acked 3656 Write 4 Done Sent Done true 2658 Acked 3657 Read 9 Done Sent Done true 2659 Acked 3658 Flush 0 Done Sent Done true 2660 Acked 3659 Read 1 Done Sent Done true 2661 Acked 3660 Read 2 Done Sent Done true 2662 Acked 3661 Write 6 Done Sent Done true 2663 Acked 3662 Write 6 Done Sent Done true 2664 Acked 3663 Read 9 Done Sent Done true 2665 Acked 3664 Write 4 Done Sent Done true 2666 Acked 3665 Read 4 Done Sent Done true 2667 Acked 3666 Write 10 Done Sent Done true 2668 Acked 3667 Write 4 Done Sent Done true 2669 Acked 3668 Write 9 Done Sent Done true 2670 Acked 3669 Write 10 Done Sent Done true 2671 Acked 3670 Write 7 Done Sent Done true 2672 Acked 3671 Write 7 Done Sent Done true 2673 Acked 3672 Read 7 Done Sent Done true 2674 Acked 3673 Read 5 Done Sent Done true 2675 Acked 3674 Write 9 Done Sent Done true 2676 Acked 3675 Read 7 Done Sent Done true 2677 Acked 3676 Read 1 Done Sent Done true 2678 Acked 3677 Read 1 Done Sent Done true 2679 Acked 3678 Write 5 Done Sent Done true 2680 Acked 3679 Read 10 Done Sent Done true 2681 Acked 3680 Read 6 Done Sent Done true 2682 Acked 3681 Write 3 Done Sent Done true 2683 Acked 3682 Write 8 Done Sent Done true 2684 Acked 3683 Read 9 Done Sent Done true 2685 Acked 3684 Write 4 Done Sent Done true 2686 Acked 3685 Read 9 Done Sent Done true 2687 Acked 3686 Write 9 Done Sent Done true 2688 Acked 3687 Read 8 Done Sent Done true 2689 Acked 3688 Read 6 Done Sent Done true 2690 Acked 3689 Read 3 Done Sent Done true 2691 Acked 3690 Write 7 Done Sent Done true 2692 Acked 3691 Read 6 Done Sent Done true 2693 Acked 3692 Write 10 Done Sent Done true 2694 Acked 3693 Flush 0 Done Sent Done true 2695 Acked 3694 Write 5 Done Sent Done true 2696 Acked 3695 Flush 0 Done Sent Done true 2697 Acked 3696 Read 9 Done Sent Done true 2698 Acked 3697 Write 5 Done Sent Done true 2699 Acked 3698 Read 2 Done Sent Done true 2700 Acked 3699 Read 6 Done Sent Done true 2701 Acked 3700 Write 6 Done Sent Done true 2702 Acked 3701 Write 7 Done Sent Done true 2703 Acked 3702 Write 6 Done Sent Done true 2704 Acked 3703 Read 10 Done Sent Done true 2705 Acked 3704 Write 6 Done Sent Done true 2706 Acked 3705 Read 8 Done Sent Done true 2707 Acked 3706 Flush 0 Done Sent Done true 2708 Acked 3707 Read 5 Done Sent Done true 2709 Acked 3708 Read 6 Done Sent Done true 2710 Acked 3709 Read 3 Done Sent Done true 2711 Acked 3710 Write 3 Done Sent Done true 2712 Acked 3711 Read 4 Done Sent Done true 2713 Acked 3712 Write 8 Done Sent Done true 2714 Acked 3713 Write 10 Done Sent Done true 2715 Acked 3714 Read 6 Done Sent Done true 2716 Acked 3715 Read 8 Done Sent Done true 2717 Acked 3716 Write 10 Done Sent Done true 2718 Acked 3717 Write 5 Done Sent Done true 2719 Acked 3718 Flush 0 Done Sent Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 218 0 218 Done 244 26 244 514 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3718 3474 3718 Downstairs last five completed: 3474 3473 3472 3471 3470 Upstairs last five completed: 2719 2718 2717 2716 2715 CLIENT: Up:0 ds:244 act:3 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 3718 3717 3716 3715 3714 Upstairs last five completed: 2719 2718 2717 2716 2715 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:32:31.100 INFO request completed, latency_us: 494, response_code: 204, uri: /stop/cid/0, [8810] Got stop action so:false kr:false method: POST, req_id: d5b14458-38a8-4b33-a9e0-8589e8cf4aec, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:32:31.101 INFO request completed, latency_us: 185, response_code: 200, uri: /state/cid/0, method: GET, req_id: 21a0640c-4b9e-4765-a6ec-4c76df1eaae4, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:31.10159002Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:31.101690007Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:31.101722785Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:31.101744922Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:31.101818139Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:31.101850197Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:32.102269791Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:32.102379272Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:32.10243991Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:32.102499252Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:32.102525613Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:32.102546103Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:32.102568637Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:32.102590839Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:33.103536283Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:33.103601237Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:33.103628843Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:33.103655362Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:33.103686007Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:33.10370986Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:33.103750833Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:34.104831173Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:34.104940472Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:34.10496917Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:34.104996086Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:34.105016652Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:34.105039372Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:34.105084413Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:32:35.104 INFO request completed, latency_us: 256, response_code: 200, uri: /state/cid/0, method: GET, req_id: 715a0dfe-bcea-4d59-a198-0bc0f1f2220c, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 523 len 40960 data: 2 2 4 4 4 4 4 4 3 3 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:35.106144794Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:35.106211678Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:35.106240033Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 002/300 Read block 4300 len 32768 {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:35.106493189Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:35.106531982Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 1 jobs since flush 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:35.106557053Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:35.10681526Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 2152 len 8192 004/300 Write block 728 len 8192 data: 4 4 005/300 Write block 57 len 20480 data: 3 3 3 3 3 006/300 Read block 1152 len 24576 007/300 Flush 008/300 Flush 009/300 Write block 3987 len 28672 data: 3 3 3 4 4 4 5 010/300 Read block 2858 len 8192 011/300 Read block 3879 len 16384 012/300 Write block 4399 len 20480 data: 3 3 3 4 3 013/300 Write block 859 len 4096 data: 3 014/300 Write block 4853 len 8192 data: 2 2 015/300 Write block 1713 len 28672 data: 3 4 4 3 3 3 3 016/300 Read block 2944 len 8192 017/300 Write block 1039 len 36864 data: 3 5 3 2 2 2 2 2 2 018/300 Read block 2019 len 36864 019/300 Flush 020/300 Write block 1711 len 12288 data: 4 3 4 021/300 Read block 2448 len 8192 022/300 Read block 4654 len 24576 023/300 Read block 1084 len 36864 024/300 Write block 3774 len 24576 data: 2 2 2 3 4 4 025/300 Flush 026/300 Write block 4462 len 4096 data: 3 027/300 Read block 1031 len 16384 028/300 Write block 4175 len 40960 data: 4 4 4 4 5 4 4 4 4 3 029/300 Write block 2391 len 24576 data: 4 4 4 3 3 3 030/300 Write block 1624 len 16384 data: 3 5 4 4 031/300 Write block 4975 len 24576 data: 3 3 2 2 3 3 032/300 Flush 033/300 Write block 4025 len 12288 data: 4 4 3 034/300 Write block 1730 len 28672 data: 3 3 3 2 2 2 4 035/300 Read block 3122 len 16384 036/300 Read block 3125 len 4096 037/300 Read block 1138 len 28672 038/300 Write block 2559 len 40960 data: 2 2 2 2 3 3 3 3 3 4 039/300 Write block 372 len 28672 data: 2 4 4 4 4 5 5 040/300 Write block 498 len 32768 data: 2 2 2 2 2 2 2 3 041/300 Write block 3513 len 28672 data: 2 2 2 2 2 2 2 042/300 Write block 1886 len 8192 data: 3 3 043/300 Read block 2663 len 36864 044/300 Read block 1034 len 32768 045/300 Read block 4201 len 28672 046/300 Read block 1310 len 12288 047/300 Write block 3324 len 12288 data: 2 2 2 048/300 Read block 325 len 40960 049/300 Flush 050/300 Read block 3622 len 36864 051/300 Read block 4456 len 8192 052/300 Write block 2632 len 36864 data: 3 3 3 3 2 2 2 2 2 053/300 Write block 2622 len 40960 data: 3 3 3 3 3 5 5 4 3 3 054/300 Write block 3857 len 32768 data: 4 3 3 3 4 5 5 5 055/300 Write block 3905 len 16384 data: 2 3 3 4 056/300 Flush 057/300 Write block 2694 len 12288 data: 4 3 3 058/300 Write block 587 len 24576 data: 3 3 3 3 2 2 059/300 Read block 4590 len 4096 060/300 Flush 061/300 Read block 1904 len 20480 062/300 Read block 4212 len 20480 063/300 Read block 3567 len 16384 064/300 Read block 1197 len 20480 065/300 Read block 1708 len 36864 066/300 Read block 1445 len 4096 067/300 Read block 3359 len 8192 068/300 Write block 319 len 24576 data: 4 3 3 3 4 3 069/300 Write block 3118 len 8192 data: 2 2 070/300 Write block 1982 len 20480 data: 3 3 3 3 4 071/300 Read block 1687 len 28672 072/300 Read block 204 len 24576 073/300 Read block 745 len 4096 074/300 Read block 3362 len 32768 075/300 Write block 3055 len 40960 data: 5 6 6 5 5 5 3 4 3 3 076/300 Read block 1097 len 36864 077/300 Read block 1377 len 36864 078/300 Read block 3051 len 12288 079/300 Read block 4300 len 36864 080/300 Flush 081/300 Read block 4323 len 16384 082/300 Read block 3150 len 8192 083/300 Read block 284 len 16384 084/300 Write block 535 len 36864 data: 4 4 4 4 4 3 3 3 3 085/300 Read block 3012 len 28672 086/300 Read block 2720 len 12288 087/300 Read block 2196 len 20480 088/300 Write block 4695 len 28672 data: 5 5 5 5 5 4 3 089/300 Write block 3939 len 8192 data: 2 2 090/300 Read block 2638 len 24576 091/300 Write block 3481 len 36864 data: 2 2 2 2 2 2 2 2 2 092/300 Read block 1719 len 12288 093/300 Read block 36 len 40960 094/300 Read block 2227 len 24576 095/300 Read block 2350 len 36864 096/300 Flush 097/300 Read block 4370 len 12288 098/300 Flush 099/300 Write block 522 len 32768 data: 2 3 3 5 5 5 5 5 100/300 Read block 4991 len 8192 101/300 Write block 685 len 28672 data: 4 4 5 5 5 5 5 102/300 Write block 4577 len 20480 data: 3 3 4 4 4 103/300 Read block 2904 len 16384 104/300 Read block 1315 len 4096 105/300 Read block 3769 len 20480 106/300 Read block 1284 len 24576 107/300 Read block 578 len 36864 108/300 Read block 4877 len 8192 109/300 Write block 216 len 28672 data: 4 4 4 3 3 3 4 110/300 Read block 3174 len 8192 111/300 Write block 4931 len 24576 data: 5 4 5 4 4 3 112/300 Write block 1882 len 20480 data: 3 3 2 3 4 113/300 Read block 4602 len 28672 114/300 Write block 2491 len 16384 data: 3 3 3 2 115/300 Read block 3405 len 28672 116/300 Write block 84 len 40960 data: 4 4 4 4 4 2 2 2 2 2 117/300 Write block 4231 len 32768 data: 5 5 5 5 4 4 3 3 118/300 Write block 426 len 24576 data: 3 3 3 3 3 4 119/300 Write block 2255 len 40960 data: 2 2 2 2 2 2 2 2 2 2 120/300 Write block 919 len 24576 data: 2 2 2 3 3 3 121/300 Write block 86 len 36864 data: 5 5 5 3 3 3 3 3 5 122/300 Read block 2582 len 24576 123/300 Read block 3838 len 28672 124/300 Write block 302 len 12288 data: 2 2 2 125/300 Write block 3649 len 12288 data: 2 2 2 126/300 Read block 2181 len 40960 127/300 Flush 128/300 Write block 2562 len 16384 data: 3 4 4 4 129/300 Read block 3116 len 28672 130/300 Read block 2740 len 8192 131/300 Flush 132/300 Read block 3799 len 28672 133/300 Write block 4307 len 4096 data: 3 134/300 Write block 1496 len 24576 data: 3 2 2 3 3 3 135/300 Read block 3744 len 4096 136/300 Write block 2696 len 16384 data: 4 2 2 2 137/300 Write block 309 len 12288 data: 5 6 6 138/300 Read block 4010 len 24576 139/300 Write block 2138 len 32768 data: 2 3 4 5 4 4 4 4 140/300 Flush 141/300 Flush 142/300 Write block 1342 len 32768 data: 4 3 4 4 4 5 5 5 143/300 Write block 2508 len 16384 data: 5 4 4 4 144/300 Read block 4374 len 16384 145/300 Write block 2015 len 16384 data: 3 3 3 2 146/300 Write block 3346 len 28672 data: 5 5 5 4 4 3 2 147/300 Write block 1660 len 20480 data: 5 5 4 3 3 148/300 Read block 2231 len 4096 149/300 Read block 1768 len 8192 150/300 Read block 2104 len 32768 151/300 Read block 2296 len 20480 152/300 Read block 3857 len 40960 153/300 Flush 154/300 Read block 4818 len 40960 155/300 Write block 2096 len 20480 data: 2 2 3 3 3 156/300 Read block 1271 len 20480 157/300 Write block 1460 len 24576 data: 2 2 3 3 3 3 158/300 Read block 727 len 12288 159/300 Read block 1522 len 24576 160/300 Write block 3190 len 24576 data: 3 3 3 3 2 2 161/300 Write block 2053 len 32768 data: 3 3 2 3 3 2 2 2 162/300 Write block 2700 len 4096 data: 2 163/300 Read block 2090 len 28672 164/300 Read block 1148 len 20480 165/300 Write block 4268 len 28672 data: 6 6 5 4 4 3 2 166/300 Flush 167/300 Write block 1332 len 8192 data: 4 5 168/300 Flush 169/300 Read block 2956 len 40960 170/300 Flush 171/300 Write block 2771 len 12288 data: 8 8 8 172/300 Flush 173/300 Flush 174/300 Write block 2900 len 8192 data: 4 4 175/300 Write block 4649 len 40960 data: 3 3 3 3 3 3 3 3 2 2 176/300 Write block 2605 len 24576 data: 3 3 2 3 3 3 177/300 Flush 178/300 Write block 2949 len 4096 data: 6 179/300 Read block 3714 len 24576 180/300 Read block 547 len 12288 181/300 Read block 1901 len 16384 182/300 Flush 183/300 Write block 2434 len 12288 data: 3 3 3 184/300 Write block 4282 len 4096 data: 3 185/300 Write block 2786 len 36864 data: 3 3 3 3 2 3 3 3 2 186/300 Read block 3347 len 16384 187/300 Write block 4418 len 20480 data: 3 3 3 3 4 188/300 Write block 1984 len 8192 data: 4 4 189/300 Read block 3619 len 16384 190/300 Read block 3857 len 28672 191/300 Read block 3494 len 12288 192/300 Write block 4626 len 4096 data: 3 193/300 Read block 3573 len 16384 194/300 Write block 4127 len 36864 data: 4 4 3 3 4 4 3 3 2 195/300 Write block 2005 len 36864 data: 4 4 6 8 8 7 6 6 4 196/300 Write block 404 len 4096 data: 3 197/300 Write block 2842 len 28672 data: 3 4 4 4 4 4 4 198/300 Read block 3311 len 20480 199/300 Read block 3540 len 28672 200/300 Read block 85 len 28672 201/300 Read block 4158 len 24576 202/300 Write block 2330 len 12288 data: 4 4 4 203/300 Write block 4445 len 8192 data: 4 6 204/300 Write block 4615 len 12288 data: 3 4 3 205/300 Write block 288 len 36864 data: 2 2 2 2 3 2 2 2 2 206/300 Read block 4395 len 8192 207/300 Flush 208/300 Read block 1485 len 40960 209/300 Read block 4619 len 32768 210/300 Read block 2387 len 28672 211/300 Flush 212/300 Write block 1294 len 16384 data: 4 4 3 3 213/300 Write block 1218 len 4096 data: 3 214/300 Write block 296 len 8192 data: 3 2 215/300 Write block 6 len 24576 data: 2 2 2 2 4 5 216/300 Write block 594 len 8192 data: 3 3 217/300 Read block 4240 len 12288 218/300 Read block 612 len 40960 219/300 Flush 220/300 Read block 2183 len 36864 221/300 Read block 1437 len 28672 222/300 Write block 2651 len 24576 data: 4 4 4 4 4 3 223/300 Read block 2939 len 36864 224/300 Read block 1328 len 8192 225/300 Read block 3442 len 32768 226/300 Write block 2383 len 20480 data: 2 2 3 3 3 227/300 Write block 4005 len 24576 data: 4 3 3 3 3 3 228/300 Write block 951 len 32768 data: 6 4 4 4 2 2 2 2 229/300 Read block 3029 len 4096 230/300 Read block 4989 len 20480 231/300 Read block 1623 len 24576 232/300 Read block 4453 len 28672 233/300 Write block 4127 len 8192 data: 5 5 234/300 Write block 1660 len 40960 data: 6 6 5 4 4 2 2 2 2 3 235/300 Read block 4340 len 32768 236/300 Read block 484 len 32768 237/300 Write block 3993 len 20480 data: 6 6 7 6 6 238/300 Read block 2753 len 36864 239/300 Write block 77 len 12288 data: 3 3 3 240/300 Write block 805 len 36864 data: 3 3 3 4 3 3 3 3 4 241/300 Read block 3780 len 32768 242/300 Read block 1231 len 32768 243/300 Read block 4834 len 4096 244/300 Read block 1886 len 24576 245/300 Write block 736 len 40960 data: 3 2 2 2 2 2 2 2 2 2 246/300 Flush 247/300 Write block 1037 len 16384 data: 2 2 4 6 248/300 Flush 249/300 Write block 3380 len 12288 data: 4 4 3 250/300 Write block 1755 len 24576 data: 4 4 4 4 4 3 251/300 Flush 252/300 Flush 253/300 Read block 1269 len 12288 254/300 Read block 3309 len 32768 255/300 Write block 923 len 32768 data: 4 4 3 3 3 3 2 2 256/300 Read block 1870 len 28672 257/300 Read block 4692 len 12288 258/300 Write block 416 len 32768 data: 3 3 4 4 4 4 4 3 259/300 Read block 4571 len 32768 260/300 Read block 273 len 8192 261/300 Write block 3625 len 40960 data: 5 4 4 4 4 3 3 3 3 3 262/300 Read block 2398 len 24576 263/300 Read block 3843 len 40960 264/300 Read block 1517 len 12288 265/300 Write block 1253 len 36864 data: 3 3 3 4 5 5 4 4 4 266/300 Write block 348 len 28672 data: 2 2 2 2 2 3 3 267/300 Read block 1523 len 28672 268/300 Read block 1981 len 36864 269/300 Read block 3543 len 32768 270/300 Read block 4202 len 12288 271/300 Read block 735 len 32768 272/300 Write block 4799 len 36864 data: 4 4 4 4 4 4 5 4 3 273/300 Write block 1748 len 8192 data: 3 3 274/300 Read block 4313 len 32768 275/300 Read block 3238 len 12288 276/300 Read block 3140 len 40960 277/300 Write block 4010 len 20480 data: 4 4 3 3 4 278/300 Read block 4068 len 12288 279/300 Write block 3881 len 24576 data: 5 6 6 5 6 7 280/300 Read block 969 len 20480 281/300 Write block 2733 len 32768 data: 5 4 4 4 4 3 3 3 282/300 Write block 275 len 40960 data: 4 3 3 3 3 3 3 3 4 4 283/300 Read block 2915 len 24576 284/300 Read block 852 len 12288 285/300 Write block 1145 len 12288 data: 2 2 2 286/300 Read block 4546 len 40960 287/300 Write block 2455 len 32768 data: 3 3 3 4 4 4 5 5 288/300 Write block 4739 len 16384 data: 5 5 4 2 289/300 Read block 3365 len 28672 290/300 Write block 3411 len 12288 data: 3 3 3 291/300 Write block 4204 len 16384 data: 3 3 3 3 292/300 Write block 234 len 16384 data: 5 4 4 3 293/300 Read block 809 len 16384 294/300 Read block 4042 len 4096 295/300 Read block 4619 len 8192 296/300 Write block 1673 len 36864 data: 3 3 4 5 4 4 4 4 4 297/300 Write block 4711 len 12288 data: 2 2 2 298/300 Write block 3428 len 20480 data: 4 3 2 2 2 299/300 Read block 1794 len 4096 300/300 Write block 223 len 20480 data: 4 4 3 3 3 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:32:35.891 INFO request completed, latency_us: 299, response_code: 204, uri: /start/cid/0, method: POST, req_id: 841564c4-d9d1-45d0-944b-956c1eb81971, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 2720 Acked 3719 Write 10 New Done Done true 2721 Acked 3720 Read 8 New Done Done false 2722 Acked 3721 Read 2 New Done Done false 2723 Acked 3722 Write 2 New Done Done false 2724 Acked 3723 Write 5 New Done Done false 2725 Acked 3724 Read 6 New Done Done false 2726 Acked 3725 Flush 0 New Done Done false 2727 Acked 3726 Flush 0 New Done Done false 2728 Acked 3727 Write 7 New Done Done false 2729 Acked 3728 Read 2 New Done Done false 2730 Acked 3729 Read 4 New Done Done false 2731 Acked 3730 Write 5 New Done Done false 2732 Acked 3731 Write 1 New Done Done false 2733 Acked 3732 Write 2 New Done Done false 2734 Acked 3733 Write 7 New Done Done false 2735 Acked 3734 Read 2 New Done Done false 2736 Acked 3735 Write 9 New Done Done false 2737 Acked 3736 Read 9 New Done Done false 2738 Acked 3737 Flush 0 New Done Done false 2739 Acked 3738 Write 3 New Done Done false 2740 Acked 3739 Read 2 New Done Done false 2741 Acked 3740 Read 6 New Done Done false 2742 Acked 3741 Read 9 New Done Done false 2743 Acked 3742 Write 6 New Done Done false 2744 Acked 3743 Flush 0 New Done Done false 2745 Acked 3744 Write 1 New Done Done false 2746 Acked 3745 Read 4 New Done Done false 2747 Acked 3746 Write 10 New Done Done false 2748 Acked 3747 Write 6 New Done Done false 2749 Acked 3748 Write 4 New Done Done false 2750 Acked 3749 Write 6 New Done Done false 2751 Acked 3750 Flush 0 New Done Done false 2752 Acked 3751 Write 3 New Done Done false 2753 Acked 3752 Write 7 New Done Done false 2754 Acked 3753 Read 4 New Done Done false 2755 Acked 3754 Read 1 New Done Done false 2756 Acked 3755 Read 7 New Done Done false 2757 Acked 3756 Write 10 New Done Done false 2758 Acked 3757 Write 7 New Done Done false 2759 Acked 3758 Write 8 New Done Done false 2760 Acked 3759 Write 7 New Done Done false 2761 Acked 3760 Write 2 New Done Done false 2762 Acked 3761 Read 9 New Done Done false 2763 Acked 3762 Read 8 New Done Done false 2764 Acked 3763 Read 7 New Done Done false 2765 Acked 3764 Read 3 New Done Done false 2766 Acked 3765 Write 3 New Done Done false 2767 Acked 3766 Read 10 New Done Done false 2768 Acked 3767 Flush 0 New Done Done false 2769 Acked 3768 Read 9 New Done Done false 2770 Acked 3769 Read 2 New Done Done false 2771 Acked 3770 Write 9 New Done Done false 2772 Acked 3771 Write 10 New Done Done false 2773 Acked 3772 Write 8 New Done Done false 2774 Acked 3773 Write 4 New Done Done false 2775 Acked 3774 Flush 0 New Done Done false 2776 Acked 3775 Write 3 New Done Done false 2777 Acked 3776 Write 6 New Done Done false 2778 Acked 3777 Read 1 New Done Done false 2779 Acked 3778 Flush 0 New Done Done false 2780 Acked 3779 Read 5 New Done Done false 2781 Acked 3780 Read 5 New Done Done false 2782 Acked 3781 Read 4 New Done Done false 2783 Acked 3782 Read 5 New Done Done false 2784 Acked 3783 Read 9 New Done Done false 2785 Acked 3784 Read 1 New Done Done false 2786 Acked 3785 Read 2 New Done Done false 2787 Acked 3786 Write 6 New Done Done false 2788 Acked 3787 Write 2 New Done Done false 2789 Acked 3788 Write 5 New Done Done false 2790 Acked 3789 Read 7 New Done Done false 2791 Acked 3790 Read 6 New Done Done false 2792 Acked 3791 Read 1 New Done Done false 2793 Acked 3792 Read 8 New Done Done false 2794 Acked 3793 Write 10 New Done Done false 2795 Acked 3794 Read 9 New Done Done false 2796 Acked 3795 Read 9 New Done Done false 2797 Acked 3796 Read 3 New Done Done false 2798 Acked 3797 Read 9 New Done Done false 2799 Acked 3798 Flush 0 New Done Done false 2800 Acked 3799 Read 4 New Done Done false 2801 Acked 3800 Read 2 New Done Done false 2802 Acked 3801 Read 4 New Done Done false 2803 Acked 3802 Write 9 New Done Done false 2804 Acked 3803 Read 7 New Done Done false 2805 Acked 3804 Read 3 New Done Done false 2806 Acked 3805 Read 5 New Done Done false 2807 Acked 3806 Write 7 New Done Done false 2808 Acked 3807 Write 2 New Done Done false 2809 Acked 3808 Read 6 New Done Done false 2810 Acked 3809 Write 9 New Done Done false 2811 Acked 3810 Read 3 New Done Done false 2812 Acked 3811 Read 10 New Done Done false 2813 Acked 3812 Read 6 New Done Done false 2814 Acked 3813 Read 9 New Done Done false 2815 Acked 3814 Flush 0 New Done Done false 2816 Acked 3815 Read 3 New Done Done false 2817 Acked 3816 Flush 0 New Done Done false 2818 Acked 3817 Write 8 New Done Done false 2819 Acked 3818 Read 2 New Done Done false 2820 Acked 3819 Write 7 New Done Done false 2821 Acked 3820 Write 5 New Done Done false 2822 Acked 3821 Read 4 New Done Done false 2823 Acked 3822 Read 1 New Done Done false 2824 Acked 3823 Read 5 New Done Done false 2825 Acked 3824 Read 6 New Done Done false 2826 Acked 3825 Read 9 New Done Done false 2827 Acked 3826 Read 2 New Done Done false 2828 Acked 3827 Write 7 New Done Done false 2829 Acked 3828 Read 2 New Done Done false 2830 Acked 3829 Write 6 New Done Done false 2831 Acked 3830 Write 5 New Done Done false 2832 Acked 3831 Read 7 New Done Done false 2833 Acked 3832 Write 4 New Done Done false 2834 Acked 3833 Read 7 New Done Done false 2835 Acked 3834 Write 10 New Done Done false 2836 Acked 3835 Write 8 New Done Done false 2837 Acked 3836 Write 6 New Done Done false 2838 Acked 3837 Write 10 New Done Done false 2839 Acked 3838 Write 6 New Done Done false 2840 Acked 3839 Write 9 New Done Done false 2841 Acked 3840 Read 6 New Done Done false 2842 Acked 3841 Read 7 New Done Done false 2843 Acked 3842 Write 3 New Done Done false 2844 Acked 3843 Write 3 New Done Done false 2845 Acked 3844 Read 10 New Done Done false 2846 Acked 3845 Flush 0 New Done Done false 2847 Acked 3846 Write 4 New Done Done false 2848 Acked 3847 Read 7 New Done Done false 2849 Acked 3848 Read 2 New Done Done false 2850 Acked 3849 Flush 0 New Done Done false 2851 Acked 3850 Read 7 New Done Done false 2852 Acked 3851 Write 1 New Done Done false 2853 Acked 3852 Write 6 New Done Done false 2854 Acked 3853 Read 1 New Done Done false 2855 Acked 3854 Write 4 New Done Done false 2856 Acked 3855 Write 3 New Done Done false 2857 Acked 3856 Read 6 New Done Done false 2858 Acked 3857 Write 8 New Done Done false 2859 Acked 3858 Flush 0 New Done Done false 2860 Acked 3859 Flush 0 New Done Done false 2861 Acked 3860 Write 8 New Done Done false 2862 Acked 3861 Write 4 New Done Done false 2863 Acked 3862 Read 4 New Done Done false 2864 Acked 3863 Write 4 New Done Done false 2865 Acked 3864 Write 7 New Done Done false 2866 Acked 3865 Write 5 New Done Done false 2867 Acked 3866 Read 1 New Done Done false 2868 Acked 3867 Read 2 New Done Done false 2869 Acked 3868 Read 8 New Done Done false 2870 Acked 3869 Read 5 New Done Done false 2871 Acked 3870 Read 10 New Done Done false 2872 Acked 3871 Flush 0 New Done Done false 2873 Acked 3872 Read 10 New Done Done false 2874 Acked 3873 Write 5 New Done Done false 2875 Acked 3874 Read 5 New Done Done false 2876 Acked 3875 Write 6 New Done Done false 2877 Acked 3876 Read 3 New Done Done false 2878 Acked 3877 Read 6 New Done Done false 2879 Acked 3878 Write 6 New Done Done false 2880 Acked 3879 Write 8 New Done Done false 2881 Acked 3880 Write 1 New Done Done false 2882 Acked 3881 Read 7 New Done Done false 2883 Acked 3882 Read 5 New Done Done false 2884 Acked 3883 Write 7 New Done Done false 2885 Acked 3884 Flush 0 New Done Done false 2886 Acked 3885 Write 2 New Done Done false 2887 Acked 3886 Flush 0 New Done Done false 2888 Acked 3887 Read 10 New Done Done false 2889 Acked 3888 Flush 0 New Done Done false 2890 Acked 3889 Flush 0 New Done Done false 2891 Acked 3890 Write 3 New Done Done false 2892 Acked 3891 Flush 0 New Done Done false 2893 Acked 3892 Flush 0 New Done Done false 2894 Acked 3893 Write 2 New Done Done false 2895 Acked 3894 Write 10 New Done Done false 2896 Acked 3895 Write 6 New Done Done false 2897 Acked 3896 Flush 0 New Done Done false 2898 Acked 3897 Write 1 New Done Done false 2899 Acked 3898 Read 6 New Done Done false 2900 Acked 3899 Read 3 New Done Done false 2901 Acked 3900 Read 4 New Done Done false 2902 Acked 3901 Flush 0 New Done Done false 2903 Acked 3902 Write 3 New Done Done false 2904 Acked 3903 Write 1 New Done Done false 2905 Acked 3904 Write 9 New Done Done false 2906 Acked 3905 Read 4 New Done Done false 2907 Acked 3906 Write 5 New Done Done false 2908 Acked 3907 Write 2 New Done Done false 2909 Acked 3908 Read 4 New Done Done false 2910 Acked 3909 Read 7 New Done Done false 2911 Acked 3910 Read 3 New Done Done false 2912 Acked 3911 Write 1 New Done Done false 2913 Acked 3912 Read 4 New Done Done false 2914 Acked 3913 Write 9 New Done Done false 2915 Acked 3914 Write 9 New Done Done false 2916 Acked 3915 Write 1 New Done Done false 2917 Acked 3916 Write 7 New Done Done false 2918 Acked 3917 Read 5 New Done Done false 2919 Acked 3918 Read 7 New Done Done false 2920 Acked 3919 Read 7 New Done Done false 2921 Acked 3920 Read 6 New Done Done false 2922 Acked 3921 Write 3 New Done Done false 2923 Acked 3922 Write 2 New Done Done false 2924 Acked 3923 Write 3 New Done Done false 2925 Acked 3924 Write 9 New Done Done false 2926 Acked 3925 Read 2 New Done Done false 2927 Acked 3926 Flush 0 New Done Done false 2928 Acked 3927 Read 10 New Done Done false 2929 Acked 3928 Read 8 New Done Done false 2930 Acked 3929 Read 7 New Done Done false 2931 Acked 3930 Flush 0 New Done Done false 2932 Acked 3931 Write 4 New Done Done false 2933 Acked 3932 Write 1 New Done Done false 2934 Acked 3933 Write 2 New Done Done false 2935 Acked 3934 Write 6 New Done Done false 2936 Acked 3935 Write 2 New Done Done false 2937 Acked 3936 Read 3 New Done Done false 2938 Acked 3937 Read 10 New Done Done false 2939 Acked 3938 Flush 0 New Done Done false 2940 Acked 3939 Read 9 New Done Done false 2941 Acked 3940 Read 7 New Done Done false 2942 Acked 3941 Write 6 New Done Done false 2943 Acked 3942 Read 9 New Done Done false 2944 Acked 3943 Read 2 New Done Done false 2945 Acked 3944 Read 8 New Done Done false 2946 Acked 3945 Write 5 New Done Done false 2947 Acked 3946 Write 6 New Done Done false 2948 Acked 3947 Write 8 New Done Done false 2949 Acked 3948 Read 1 New Done Done false 2950 Acked 3949 Read 5 New Done Done false 2951 Acked 3950 Read 6 New Done Done false 2952 Acked 3951 Read 7 New Done Done false 2953 Acked 3952 Write 2 New Done Done false 2954 Acked 3953 Write 10 New Done Done false 2955 Acked 3954 Read 8 New Done Done false 2956 Acked 3955 Read 8 New Done Done false 2957 Acked 3956 Write 5 New Done Done false 2958 Acked 3957 Read 9 New Done Done false 2959 Acked 3958 Write 3 New Done Done false 2960 Acked 3959 Write 9 New Done Done false 2961 Acked 3960 Read 8 New Done Done false 2962 Acked 3961 Read 8 New Done Done false 2963 Acked 3962 Read 1 New Done Done false 2964 Acked 3963 Read 6 New Done Done false 2965 Acked 3964 Write 10 New Done Done false 2966 Acked 3965 Flush 0 New Done Done false 2967 Acked 3966 Write 4 New Done Done false 2968 Acked 3967 Flush 0 New Done Done false 2969 Acked 3968 Write 3 New Done Done false 2970 Acked 3969 Write 6 New Done Done false 2971 Acked 3970 Flush 0 New Done Done false 2972 Acked 3971 Flush 0 New Done Done false 2973 Acked 3972 Read 3 New Done Done false 2974 Acked 3973 Read 8 New Done Done false 2975 Acked 3974 Write 8 New Done Done false 2976 Acked 3975 Read 7 New Done Done false 2977 Acked 3976 Read 3 New Done Done false 2978 Acked 3977 Write 8 New Done Done false 2979 Acked 3978 Read 8 New Done Done false 2980 Acked 3979 Read 2 New Done Done false 2981 Acked 3980 Write 10 New Done Done false 2982 Acked 3981 Read 6 New Done Done false 2983 Acked 3982 Read 10 New Done Done false 2984 Acked 3983 Read 3 New Done Done false 2985 Acked 3984 Write 9 New Done Done false 2986 Acked 3985 Write 7 New Done Done false 2987 Acked 3986 Read 7 New Done Done false 2988 Acked 3987 Read 9 New Done Done false 2989 Acked 3988 Read 8 New Done Done false 2990 Acked 3989 Read 3 New Done Done false 2991 Acked 3990 Read 8 New Done Done false 2992 Acked 3991 Write 9 New Done Done false 2993 Acked 3992 Write 2 New Done Done false 2994 Acked 3993 Read 8 New Done Done false 2995 Acked 3994 Read 3 New Done Done false 2996 Acked 3995 Read 10 New Done Done false 2997 Acked 3996 Write 5 New Done Done false 2998 Acked 3997 Read 3 New Done Done false 2999 Acked 3998 Write 6 New Done Done false 3000 Acked 3999 Read 5 New Done Done false 3001 Acked 4000 Write 8 New Done Done false 3002 Acked 4001 Write 10 New Done Done false 3003 Acked 4002 Read 6 New Done Done false 3004 Acked 4003 Read 3 New Done Done false 3005 Acked 4004 Write 3 New Done Done false 3006 Acked 4005 Read 10 New Done Done false 3007 Acked 4006 Write 8 New Done Done false 3008 Acked 4007 Write 4 New Done Done false 3009 Acked 4008 Read 7 New Done Done false 3010 Acked 4009 Write 3 New Done Done false 3011 Acked 4010 Write 4 New Done Done false 3012 Acked 4011 Write 4 New Done Done false 3013 Acked 4012 Read 4 New Done Done false 3014 Acked 4013 Read 1 New Done Done false 3015 Acked 4014 Read 2 New Done Done false 3016 Acked 4015 Write 9 New Done Done false 3017 Acked 4016 Write 3 New Done Done false 3018 Acked 4017 Write 5 New Done Done false 3019 Acked 4018 Read 1 New Done Done false 3020 Acked 4019 Write 5 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 3718 3971 3971 Downstairs last five completed: 3718 3717 3716 3715 3714 Upstairs last five completed: 3020 3019 3018 3017 3016 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:36.107862492Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:36.107931734Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:36.107960778Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:36.107987784Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:36.1080086Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:36.108031446Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:36.108266843Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:37.109094181Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:37.109161302Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:37.109189144Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:37.109213941Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:37.109239947Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:37.109260431Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:37.109282841Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:37.109477655Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1195) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.11037089Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.11050526Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.112544182Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.112579303Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.112603159Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 3718","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.1129205Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.112955339Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.112978889Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.113001357Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:38.113023602Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4020 4019 4018 4017 4016 Upstairs last five completed: 3021 3020 3019 3018 3017 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(2) stop 2 Jan 02 22:32:39.896 INFO request completed, latency_us: 484, response_code: 204, uri: /stop/cid/2, method: [8830] Got stop action so:false kr:false POST, req_id: f43d4b4d-caca-4f38-9502-444bcbefbb9a, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:32:39.897 INFO request completed, latency_us: 183, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0def5a0f-9a08-4f9d-9b86-b034cf7c1f16, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:39.897417326Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:39.897540649Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:39.897582332Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:39.897604788Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:39.897722574Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:39.897914095Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:40.898812295Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:40.898888144Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:40.898917098Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:40.898943801Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:40.898964424Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:40.898998972Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:40.899098153Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:41.901050372Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:41.901119373Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:41.901146969Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:41.901171159Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:41.901197414Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:41.901217744Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:41.901311302Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:41.901459468Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:42.902312565Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:42.902378201Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:42.902405927Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:42.902432489Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:42.902452777Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:42.902559468Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:42.902604974Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:32:43.899 INFO request completed, latency_us: 258, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1f2ce268-1a29-4b7d-8709-eaa2d60751e8, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 1423 len 32768 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:43.903210763Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:43.903279203Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:43.903308244Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:43.903334936Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:43.903363626Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 1 jobs since flush 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:43.903386997Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:43.903425543Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 002/300 Read block 2507 len 12288 003/300 Read block 2355 len 12288 004/300 Read block 2577 len 24576 005/300 Write block 3954 len 20480 data: 3 3 3 3 5 006/300 Read block 3683 len 4096 007/300 Write block 159 len 8192 data: 4 5 008/300 Read block 1238 len 20480 009/300 Write block 1127 len 24576 data: 4 3 3 2 3 4 010/300 Write block 812 len 4096 data: 4 011/300 Write block 4363 len 36864 data: 3 3 3 2 2 2 2 2 2 012/300 Read block 3564 len 8192 013/300 Write block 3598 len 12288 data: 3 2 3 014/300 Read block 1555 len 4096 015/300 Read block 1898 len 24576 016/300 Write block 98 len 40960 data: 4 4 4 4 2 4 4 5 5 3 017/300 Write block 2532 len 40960 data: 3 3 3 3 3 2 2 2 2 2 018/300 Write block 4141 len 16384 data: 3 2 4 5 019/300 Write block 4443 len 40960 data: 5 5 5 7 6 6 3 3 2 2 020/300 Write block 311 len 36864 data: 7 6 7 6 5 4 3 4 5 021/300 Write block 4442 len 4096 data: 5 022/300 Read block 4204 len 40960 023/300 Write block 4965 len 20480 data: 3 2 2 2 2 024/300 Write block 1364 len 24576 data: 3 3 3 2 2 3 025/300 Read block 558 len 16384 026/300 Read block 2751 len 32768 027/300 Read block 1591 len 20480 028/300 Write block 2703 len 16384 data: 2 2 2 2 029/300 Read block 4052 len 40960 030/300 Write block 11 len 40960 data: 6 4 4 5 6 6 5 5 4 4 031/300 Write block 4938 len 4096 data: 3 032/300 Write block 1109 len 4096 data: 5 033/300 Flush 034/300 Read block 2720 len 20480 035/300 Flush 036/300 Write block 275 len 36864 data: 5 4 4 4 4 4 4 4 5 037/300 Flush 038/300 Read block 1190 len 28672 039/300 Read block 4674 len 16384 040/300 Read block 1998 len 20480 041/300 Read block 3975 len 16384 042/300 Write block 1332 len 40960 data: 5 6 5 4 3 3 3 3 3 4 043/300 Write block 1766 len 28672 data: 3 3 4 4 4 4 5 044/300 Read block 324 len 24576 045/300 Read block 2148 len 20480 046/300 Read block 3762 len 8192 047/300 Read block 4456 len 24576 048/300 Write block 2806 len 4096 data: 3 049/300 Write block 4575 len 32768 data: 3 3 4 4 5 5 5 4 050/300 Read block 4223 len 20480 051/300 Write block 3723 len 36864 data: 5 5 5 5 4 4 4 4 4 052/300 Read block 2422 len 32768 053/300 Read block 4683 len 16384 054/300 Write block 658 len 32768 data: 5 5 5 4 4 6 6 6 055/300 Write block 2264 len 20480 data: 3 2 3 3 4 056/300 Write block 4757 len 12288 data: 4 3 3 057/300 Write block 2947 len 20480 data: 4 6 7 6 6 058/300 Read block 4108 len 32768 059/300 Write block 306 len 28672 data: 2 2 4 6 7 8 7 060/300 Read block 547 len 12288 061/300 Read block 1658 len 20480 062/300 Read block 531 len 12288 063/300 Read block 1107 len 40960 064/300 Read block 797 len 16384 065/300 Write block 979 len 8192 data: 2 2 066/300 Write block 3678 len 20480 data: 3 2 2 2 2 067/300 Write block 2340 len 4096 data: 3 068/300 Flush 069/300 Read block 3983 len 28672 070/300 Write block 1625 len 8192 data: 6 5 071/300 Flush 072/300 Read block 4500 len 36864 073/300 Read block 281 len 24576 074/300 Write block 1428 len 12288 data: 2 2 2 075/300 Write block 4746 len 20480 data: 2 3 3 3 3 076/300 Read block 3608 len 36864 077/300 Read block 812 len 20480 078/300 Read block 2769 len 28672 079/300 Read block 3578 len 8192 080/300 Write block 2932 len 20480 data: 2 3 4 3 4 081/300 Read block 1250 len 40960 082/300 Flush 083/300 Read block 76 len 20480 084/300 Read block 3293 len 40960 085/300 Read block 625 len 32768 086/300 Read block 738 len 32768 087/300 Read block 3750 len 28672 088/300 Read block 2140 len 40960 089/300 Write block 1 len 24576 data: 2 2 2 2 2 3 090/300 Write block 436 len 40960 data: 4 7 8 8 8 6 6 5 4 5 091/300 Write block 321 len 8192 data: 4 4 092/300 Read block 1535 len 12288 093/300 Write block 3102 len 32768 data: 4 3 3 3 4 3 2 2 094/300 Flush 095/300 Write block 1556 len 28672 data: 3 3 4 3 3 3 2 096/300 Write block 756 len 8192 data: 4 4 097/300 Read block 3977 len 8192 098/300 Read block 243 len 8192 099/300 Write block 604 len 32768 data: 3 3 3 3 3 3 3 3 100/300 Read block 3044 len 36864 101/300 Read block 2288 len 40960 102/300 Read block 3034 len 16384 103/300 Write block 3013 len 12288 data: 4 5 5 104/300 Read block 3430 len 32768 105/300 Write block 3529 len 40960 data: 4 4 4 4 3 3 2 3 5 5 106/300 Read block 4545 len 32768 107/300 Flush 108/300 Read block 3729 len 16384 109/300 Read block 4314 len 28672 110/300 Flush 111/300 Read block 832 len 12288 112/300 Write block 4566 len 28672 data: 3 3 3 3 3 3 3 113/300 Write block 1130 len 4096 data: 3 114/300 Write block 3764 len 20480 data: 5 6 8 6 6 115/300 Read block 4329 len 20480 116/300 Write block 1665 len 28672 data: 3 3 3 3 4 3 3 117/300 Write block 4716 len 40960 data: 3 3 3 3 3 2 3 3 4 4 118/300 Write block 2510 len 16384 data: 5 5 3 3 119/300 Flush 120/300 Write block 1598 len 36864 data: 3 3 2 2 2 2 2 2 2 121/300 Read block 3716 len 28672 122/300 Read block 4714 len 20480 123/300 Read block 1064 len 24576 124/300 Read block 2890 len 8192 125/300 Write block 4606 len 36864 data: 5 4 4 5 5 5 4 4 4 126/300 Read block 4683 len 28672 127/300 Flush 128/300 Flush 129/300 Read block 3033 len 4096 130/300 Read block 1533 len 36864 131/300 Read block 4233 len 12288 132/300 Write block 4194 len 8192 data: 3 2 133/300 Write block 3589 len 20480 data: 4 5 6 5 5 134/300 Read block 3161 len 24576 135/300 Write block 4517 len 8192 data: 4 4 136/300 Flush 137/300 Write block 724 len 40960 data: 2 3 3 4 5 5 4 5 5 4 138/300 Write block 27 len 12288 data: 4 4 3 139/300 Write block 1021 len 24576 data: 3 3 3 2 2 2 140/300 Read block 2167 len 28672 141/300 Read block 1986 len 32768 142/300 Read block 3639 len 8192 143/300 Read block 2355 len 16384 144/300 Write block 3502 len 36864 data: 3 3 3 3 2 2 2 2 2 145/300 Write block 1901 len 12288 data: 2 2 2 146/300 Read block 919 len 40960 147/300 Read block 1935 len 36864 148/300 Read block 1818 len 32768 149/300 Flush 150/300 Read block 1696 len 36864 151/300 Read block 3868 len 32768 152/300 Read block 3375 len 28672 153/300 Write block 1187 len 28672 data: 2 2 2 2 2 2 2 154/300 Write block 793 len 20480 data: 3 3 4 4 4 155/300 Read block 4094 len 36864 156/300 Write block 4315 len 20480 data: 2 2 3 4 4 157/300 Write block 1999 len 12288 data: 3 3 4 158/300 Write block 4976 len 32768 data: 4 3 3 4 4 3 3 5 159/300 Read block 144 len 32768 160/300 Read block 4101 len 12288 161/300 Read block 1254 len 8192 162/300 Write block 2010 len 4096 data: 8 163/300 Read block 3889 len 32768 164/300 Read block 3843 len 28672 165/300 Write block 4393 len 12288 data: 2 2 2 166/300 Write block 2892 len 8192 data: 3 3 167/300 Write block 1300 len 12288 data: 4 3 3 168/300 Read block 4378 len 24576 169/300 Read block 481 len 12288 170/300 Read block 1976 len 8192 171/300 Read block 484 len 32768 172/300 Write block 2948 len 36864 data: 7 8 7 7 5 5 4 3 3 173/300 Write block 4320 len 24576 data: 5 5 5 5 5 5 174/300 Read block 4590 len 28672 175/300 Read block 418 len 16384 176/300 Read block 2319 len 36864 177/300 Write block 2276 len 12288 data: 4 4 3 178/300 Flush 179/300 Read block 2121 len 32768 180/300 Read block 2294 len 40960 181/300 Flush 182/300 Flush 183/300 Write block 2454 len 4096 data: 2 184/300 Read block 3769 len 40960 185/300 Write block 842 len 36864 data: 2 3 3 3 4 5 5 5 5 186/300 Write block 4598 len 32768 data: 2 2 2 3 3 4 4 5 187/300 Read block 72 len 32768 188/300 Write block 2746 len 12288 data: 3 3 3 189/300 Read block 460 len 16384 190/300 Read block 3307 len 12288 191/300 Read block 631 len 32768 192/300 Read block 2485 len 16384 193/300 Flush 194/300 Write block 2947 len 4096 data: 5 195/300 Write block 1379 len 16384 data: 4 2 2 2 196/300 Write block 4627 len 16384 data: 3 3 4 4 197/300 Flush 198/300 Write block 2256 len 4096 data: 3 199/300 Read block 4050 len 20480 200/300 Write block 4333 len 12288 data: 3 3 4 201/300 Read block 2562 len 28672 202/300 Flush 203/300 Write block 1705 len 12288 data: 2 2 2 204/300 Read block 1757 len 40960 205/300 Write block 4626 len 32768 data: 4 4 4 5 5 4 4 5 206/300 Write block 2804 len 24576 data: 4 3 4 3 3 3 207/300 Write block 2122 len 8192 data: 4 4 208/300 Write block 4745 len 4096 data: 2 209/300 Write block 363 len 28672 data: 4 4 3 4 5 2 2 210/300 Flush 211/300 Read block 2897 len 36864 212/300 Write block 1113 len 36864 data: 5 5 5 3 2 2 2 2 3 213/300 Read block 2861 len 16384 214/300 Write block 3225 len 24576 data: 4 4 3 4 4 4 215/300 Read block 3147 len 36864 216/300 Write block 4047 len 12288 data: 5 4 4 217/300 Read block 3203 len 32768 218/300 Write block 2759 len 28672 data: 2 2 2 2 2 4 5 219/300 Write block 4547 len 12288 data: 3 3 2 220/300 Read block 3293 len 40960 221/300 Read block 3169 len 24576 222/300 Write block 1036 len 4096 data: 2 223/300 Read block 274 len 8192 224/300 Write block 1883 len 16384 data: 4 3 4 5 225/300 Flush 226/300 Read block 2710 len 4096 227/300 Write block 270 len 12288 data: 5 6 6 228/300 Flush 229/300 Write block 393 len 36864 data: 4 4 4 2 2 3 4 3 3 230/300 Read block 1313 len 32768 231/300 Write block 4256 len 28672 data: 4 4 3 3 4 4 4 232/300 Write block 3008 len 12288 data: 2 2 3 233/300 Write block 290 len 36864 data: 3 3 4 3 3 3 4 3 2 234/300 Write block 2226 len 32768 data: 2 3 3 3 3 3 2 3 235/300 Write block 3470 len 20480 data: 3 2 2 2 3 236/300 Write block 2256 len 28672 data: 4 3 3 3 3 3 3 237/300 Flush 238/300 Read block 4580 len 12288 239/300 Write block 2769 len 12288 data: 7 9 9 240/300 Flush 241/300 Read block 3880 len 12288 242/300 Read block 2487 len 4096 243/300 Flush 244/300 Read block 1139 len 32768 245/300 Write block 2305 len 16384 data: 3 3 3 3 246/300 Read block 489 len 4096 247/300 Read block 344 len 8192 248/300 Flush 249/300 Read block 4435 len 36864 250/300 Write block 4533 len 40960 data: 2 4 5 5 4 5 5 5 5 5 251/300 Read block 4670 len 20480 252/300 Read block 1476 len 36864 253/300 Read block 1960 len 4096 254/300 Flush 255/300 Read block 3248 len 36864 256/300 Write block 639 len 16384 data: 3 3 3 3 257/300 Write block 1772 len 20480 data: 6 5 4 6 5 258/300 Read block 2591 len 16384 259/300 Write block 2965 len 24576 data: 5 4 4 4 4 3 260/300 Flush 261/300 Write block 4472 len 16384 data: 2 2 2 2 262/300 Read block 3489 len 32768 263/300 Read block 27 len 20480 264/300 Read block 4892 len 4096 265/300 Write block 4003 len 28672 data: 5 4 5 4 4 4 4 266/300 Read block 3580 len 20480 267/300 Read block 878 len 24576 268/300 Read block 2486 len 32768 269/300 Write block 84 len 8192 data: 5 5 270/300 Flush 271/300 Write block 583 len 16384 data: 4 3 3 3 272/300 Write block 3164 len 12288 data: 2 2 2 273/300 Write block 2851 len 36864 data: 2 2 3 3 3 3 4 4 4 274/300 Read block 217 len 32768 275/300 Write block 1692 len 28672 data: 2 2 3 3 3 3 3 276/300 Write block 294 len 32768 data: 4 4 5 4 3 2 2 2 277/300 Flush 278/300 Read block 2507 len 28672 279/300 Read block 2631 len 16384 280/300 Write block 4551 len 16384 data: 2 2 2 2 281/300 Read block 1897 len 40960 282/300 Write block 945 len 16384 data: 3 4 3 4 283/300 Read block 3913 len 36864 284/300 Write block 337 len 12288 data: 5 6 6 285/300 Read block 1835 len 36864 286/300 Write block 2137 len 12288 data: 2 3 4 287/300 Read block 4472 len 28672 288/300 Write block 4651 len 32768 data: 4 4 4 4 4 4 3 3 289/300 Write block 1966 len 4096 data: 3 290/300 Write block 4903 len 20480 data: 4 4 3 3 4 291/300 Write block 4489 len 24576 data: 2 2 2 2 2 2 292/300 Read block 2751 len 20480 293/300 Write block 2398 len 20480 data: 2 3 3 3 3 294/300 Read block 2983 len 40960 295/300 Write block 4143 len 24576 data: 5 6 6 5 4 4 296/300 Read block 835 len 36864 297/300 Read block 337 len 40960 298/300 Read block 4754 len 36864 299/300 Read block 4260 len 32768 300/300 Read block 2300 len 4096 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:32:44.508 INFO request completed, latency_us: 286, response_code: 204, uri: /start/cid/2, method: POST, req_id: f618e652-8d83-4fe2-8739-3c51025330db, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 3022 Acked 4021 Read 8 Done Done New true 3023 Acked 4022 Read 3 Done Done New false 3024 Acked 4023 Read 3 Done Done New false 3025 Acked 4024 Read 6 Done Done New false 3026 Acked 4025 Write 5 Done Done New false 3027 Acked 4026 Read 1 Done Done New false 3028 Acked 4027 Write 2 Done Done New false 3029 Acked 4028 Read 5 Done Done New false 3030 Acked 4029 Write 6 Done Done New false 3031 Acked 4030 Write 1 Done Done New false 3032 Acked 4031 Write 9 Done Done New false 3033 Acked 4032 Read 2 Done Done New false 3034 Acked 4033 Write 3 Done Done New false 3035 Acked 4034 Read 1 Done Done New false 3036 Acked 4035 Read 6 Done Done New false 3037 Acked 4036 Write 10 Done Done New false 3038 Acked 4037 Write 10 Done Done New false 3039 Acked 4038 Write 4 Done Done New false 3040 Acked 4039 Write 10 Done Done New false 3041 Acked 4040 Write 9 Done Done New false 3042 Acked 4041 Write 1 Done Done New false 3043 Acked 4042 Read 10 Done Done New false 3044 Acked 4043 Write 5 Done Done New false 3045 Acked 4044 Write 6 Done Done New false 3046 Acked 4045 Read 4 Done Done New false 3047 Acked 4046 Read 8 Done Done New false 3048 Acked 4047 Read 5 Done Done New false 3049 Acked 4048 Write 4 Done Done New false 3050 Acked 4049 Read 10 Done Done New false 3051 Acked 4050 Write 10 Done Done New false 3052 Acked 4051 Write 1 Done Done New false 3053 Acked 4052 Write 1 Done Done New false 3054 Acked 4053 Flush 0 Done Done New false 3055 Acked 4054 Read 5 Done Done New false 3056 Acked 4055 Flush 0 Done Done New false 3057 Acked 4056 Write 9 Done Done New false 3058 Acked 4057 Flush 0 Done Done New false 3059 Acked 4058 Read 7 Done Done New false 3060 Acked 4059 Read 4 Done Done New false 3061 Acked 4060 Read 5 Done Done New false 3062 Acked 4061 Read 4 Done Done New false 3063 Acked 4062 Write 10 Done Done New false 3064 Acked 4063 Write 7 Done Done New false 3065 Acked 4064 Read 6 Done Done New false 3066 Acked 4065 Read 5 Done Done New false 3067 Acked 4066 Read 2 Done Done New false 3068 Acked 4067 Read 6 Done Done New false 3069 Acked 4068 Write 1 Done Done New false 3070 Acked 4069 Write 8 Done Done New false 3071 Acked 4070 Read 5 Done Done New false 3072 Acked 4071 Write 9 Done Done New false 3073 Acked 4072 Read 8 Done Done New false 3074 Acked 4073 Read 4 Done Done New false 3075 Acked 4074 Write 8 Done Done New false 3076 Acked 4075 Write 5 Done Done New false 3077 Acked 4076 Write 3 Done Done New false 3078 Acked 4077 Write 5 Done Done New false 3079 Acked 4078 Read 8 Done Done New false 3080 Acked 4079 Write 7 Done Done New false 3081 Acked 4080 Read 3 Done Done New false 3082 Acked 4081 Read 5 Done Done New false 3083 Acked 4082 Read 3 Done Done New false 3084 Acked 4083 Read 10 Done Done New false 3085 Acked 4084 Read 4 Done Done New false 3086 Acked 4085 Write 2 Done Done New false 3087 Acked 4086 Write 5 Done Done New false 3088 Acked 4087 Write 1 Done Done New false 3089 Acked 4088 Flush 0 Done Done New false 3090 Acked 4089 Read 7 Done Done New false 3091 Acked 4090 Write 2 Done Done New false 3092 Acked 4091 Flush 0 Done Done New false 3093 Acked 4092 Read 9 Done Done New false 3094 Acked 4093 Read 6 Done Done New false 3095 Acked 4094 Write 3 Done Done New false 3096 Acked 4095 Write 5 Done Done New false 3097 Acked 4096 Read 9 Done Done New false 3098 Acked 4097 Read 5 Done Done New false 3099 Acked 4098 Read 7 Done Done New false 3100 Acked 4099 Read 2 Done Done New false 3101 Acked 4100 Write 5 Done Done New false 3102 Acked 4101 Flush 0 Done Done New false 3103 Acked 4102 Read 10 Done Done New false 3104 Acked 4103 Flush 0 Done Done New false 3105 Acked 4104 Read 5 Done Done New false 3106 Acked 4105 Read 10 Done Done New false 3107 Acked 4106 Read 8 Done Done New false 3108 Acked 4107 Read 8 Done Done New false 3109 Acked 4108 Read 7 Done Done New false 3110 Acked 4109 Read 10 Done Done New false 3111 Acked 4110 Write 6 Done Done New false 3112 Acked 4111 Write 10 Done Done New false 3113 Acked 4112 Write 2 Done Done New false 3114 Acked 4113 Read 3 Done Done New false 3115 Acked 4114 Write 8 Done Done New false 3116 Acked 4115 Flush 0 Done Done New false 3117 Acked 4116 Write 7 Done Done New false 3118 Acked 4117 Write 2 Done Done New false 3119 Acked 4118 Read 2 Done Done New false 3120 Acked 4119 Read 2 Done Done New false 3121 Acked 4120 Write 8 Done Done New false 3122 Acked 4121 Read 9 Done Done New false 3123 Acked 4122 Read 10 Done Done New false 3124 Acked 4123 Read 4 Done Done New false 3125 Acked 4124 Write 3 Done Done New false 3126 Acked 4125 Read 8 Done Done New false 3127 Acked 4126 Write 10 Done Done New false 3128 Acked 4127 Read 8 Done Done New false 3129 Acked 4128 Flush 0 Done Done New false 3130 Acked 4129 Read 4 Done Done New false 3131 Acked 4130 Read 7 Done Done New false 3132 Acked 4131 Flush 0 Done Done New false 3133 Acked 4132 Read 3 Done Done New false 3134 Acked 4133 Write 7 Done Done New false 3135 Acked 4134 Write 1 Done Done New false 3136 Acked 4135 Write 5 Done Done New false 3137 Acked 4136 Read 5 Done Done New false 3138 Acked 4137 Write 7 Done Done New false 3139 Acked 4138 Write 10 Done Done New false 3140 Acked 4139 Write 4 Done Done New false 3141 Acked 4140 Flush 0 Done Done New false 3142 Acked 4141 Write 9 Done Done New false 3143 Acked 4142 Read 7 Done Done New false 3144 Acked 4143 Read 5 Done Done New false 3145 Acked 4144 Read 6 Done Done New false 3146 Acked 4145 Read 2 Done Done New false 3147 Acked 4146 Write 9 Done Done New false 3148 Acked 4147 Read 7 Done Done New false 3149 Acked 4148 Flush 0 Done Done New false 3150 Acked 4149 Flush 0 Done Done New false 3151 Acked 4150 Read 1 Done Done New false 3152 Acked 4151 Read 9 Done Done New false 3153 Acked 4152 Read 3 Done Done New false 3154 Acked 4153 Write 2 Done Done New false 3155 Acked 4154 Write 5 Done Done New false 3156 Acked 4155 Read 6 Done Done New false 3157 Acked 4156 Write 2 Done Done New false 3158 Acked 4157 Flush 0 Done Done New false 3159 Acked 4158 Write 10 Done Done New false 3160 Acked 4159 Write 3 Done Done New false 3161 Acked 4160 Write 6 Done Done New false 3162 Acked 4161 Read 7 Done Done New false 3163 Acked 4162 Read 8 Done Done New false 3164 Acked 4163 Read 2 Done Done New false 3165 Acked 4164 Read 4 Done Done New false 3166 Acked 4165 Write 9 Done Done New false 3167 Acked 4166 Write 3 Done Done New false 3168 Acked 4167 Read 10 Done Done New false 3169 Acked 4168 Read 9 Done Done New false 3170 Acked 4169 Read 8 Done Done New false 3171 Acked 4170 Flush 0 Done Done New false 3172 Acked 4171 Read 9 Done Done New false 3173 Acked 4172 Read 8 Done Done New false 3174 Acked 4173 Read 7 Done Done New false 3175 Acked 4174 Write 7 Done Done New false 3176 Acked 4175 Write 5 Done Done New false 3177 Acked 4176 Read 9 Done Done New false 3178 Acked 4177 Write 5 Done Done New false 3179 Acked 4178 Write 3 Done Done New false 3180 Acked 4179 Write 8 Done Done New false 3181 Acked 4180 Read 8 Done Done New false 3182 Acked 4181 Read 3 Done Done New false 3183 Acked 4182 Read 2 Done Done New false 3184 Acked 4183 Write 1 Done Done New false 3185 Acked 4184 Read 8 Done Done New false 3186 Acked 4185 Read 7 Done Done New false 3187 Acked 4186 Write 3 Done Done New false 3188 Acked 4187 Write 2 Done Done New false 3189 Acked 4188 Write 3 Done Done New false 3190 Acked 4189 Read 6 Done Done New false 3191 Acked 4190 Read 3 Done Done New false 3192 Acked 4191 Read 2 Done Done New false 3193 Acked 4192 Read 8 Done Done New false 3194 Acked 4193 Write 9 Done Done New false 3195 Acked 4194 Write 6 Done Done New false 3196 Acked 4195 Read 7 Done Done New false 3197 Acked 4196 Read 4 Done Done New false 3198 Acked 4197 Read 9 Done Done New false 3199 Acked 4198 Write 3 Done Done New false 3200 Acked 4199 Flush 0 Done Done New false 3201 Acked 4200 Read 8 Done Done New false 3202 Acked 4201 Read 10 Done Done New false 3203 Acked 4202 Flush 0 Done Done New false 3204 Acked 4203 Flush 0 Done Done New false 3205 Acked 4204 Write 1 Done Done New false 3206 Acked 4205 Read 10 Done Done New false 3207 Acked 4206 Write 9 Done Done New false 3208 Acked 4207 Write 8 Done Done New false 3209 Acked 4208 Read 8 Done Done New false 3210 Acked 4209 Write 3 Done Done New false 3211 Acked 4210 Read 4 Done Done New false 3212 Acked 4211 Read 3 Done Done New false 3213 Acked 4212 Read 8 Done Done New false 3214 Acked 4213 Read 4 Done Done New false 3215 Acked 4214 Flush 0 Done Done New false 3216 Acked 4215 Write 1 Done Done New false 3217 Acked 4216 Write 4 Done Done New false 3218 Acked 4217 Write 4 Done Done New false 3219 Acked 4218 Flush 0 Done Done New false 3220 Acked 4219 Write 1 Done Done New false 3221 Acked 4220 Read 5 Done Done New false 3222 Acked 4221 Write 3 Done Done New false 3223 Acked 4222 Read 7 Done Done New false 3224 Acked 4223 Flush 0 Done Done New false 3225 Acked 4224 Write 3 Done Done New false 3226 Acked 4225 Read 10 Done Done New false 3227 Acked 4226 Write 8 Done Done New false 3228 Acked 4227 Write 6 Done Done New false 3229 Acked 4228 Write 2 Done Done New false 3230 Acked 4229 Write 1 Done Done New false 3231 Acked 4230 Write 7 Done Done New false 3232 Acked 4231 Flush 0 Done Done New false 3233 Acked 4232 Read 9 Done Done New false 3234 Acked 4233 Write 9 Done Done New false 3235 Acked 4234 Read 4 Done Done New false 3236 Acked 4235 Write 6 Done Done New false 3237 Acked 4236 Read 9 Done Done New false 3238 Acked 4237 Write 3 Done Done New false 3239 Acked 4238 Read 8 Done Done New false 3240 Acked 4239 Write 7 Done Done New false 3241 Acked 4240 Write 3 Done Done New false 3242 Acked 4241 Read 10 Done Done New false 3243 Acked 4242 Read 6 Done Done New false 3244 Acked 4243 Write 1 Done Done New false 3245 Acked 4244 Read 2 Done Done New false 3246 Acked 4245 Write 4 Done Done New false 3247 Acked 4246 Flush 0 Done Done New false 3248 Acked 4247 Read 1 Done Done New false 3249 Acked 4248 Write 3 Done Done New false 3250 Acked 4249 Flush 0 Done Done New false 3251 Acked 4250 Write 9 Done Done New false 3252 Acked 4251 Read 8 Done Done New false 3253 Acked 4252 Write 7 Done Done New false 3254 Acked 4253 Write 3 Done Done New false 3255 Acked 4254 Write 9 Done Done New false 3256 Acked 4255 Write 8 Done Done New false 3257 Acked 4256 Write 5 Done Done New false 3258 Acked 4257 Write 7 Done Done New false 3259 Acked 4258 Flush 0 Done Done New false 3260 Acked 4259 Read 3 Done Done New false 3261 Acked 4260 Write 3 Done Done New false 3262 Acked 4261 Flush 0 Done Done New false 3263 Acked 4262 Read 3 Done Done New false 3264 Acked 4263 Read 1 Done Done New false 3265 Acked 4264 Flush 0 Done Done New false 3266 Acked 4265 Read 8 Done Done New false 3267 Acked 4266 Write 4 Done Done New false 3268 Acked 4267 Read 1 Done Done New false 3269 Acked 4268 Read 2 Done Done New false 3270 Acked 4269 Flush 0 Done Done New false 3271 Acked 4270 Read 9 Done Done New false 3272 Acked 4271 Write 10 Done Done New false 3273 Acked 4272 Read 5 Done Done New false 3274 Acked 4273 Read 9 Done Done New false 3275 Acked 4274 Read 1 Done Done New false 3276 Acked 4275 Flush 0 Done Done New false 3277 Acked 4276 Read 9 Done Done New false 3278 Acked 4277 Write 4 Done Done New false 3279 Acked 4278 Write 5 Done Done New false 3280 Acked 4279 Read 4 Done Done New false 3281 Acked 4280 Write 6 Done Done New false 3282 Acked 4281 Flush 0 Done Done New false 3283 Acked 4282 Write 4 Done Done New false 3284 Acked 4283 Read 8 Done Done New false 3285 Acked 4284 Read 5 Done Done New false 3286 Acked 4285 Read 1 Done Done New false 3287 Acked 4286 Write 7 Done Done New false 3288 Acked 4287 Read 5 Done Done New false 3289 Acked 4288 Read 6 Done Done New false 3290 Acked 4289 Read 8 Done Done New false 3291 Acked 4290 Write 2 Done Done New false 3292 Acked 4291 Flush 0 Done Done New false 3293 Acked 4292 Write 4 Done Done New false 3294 Acked 4293 Write 3 Done Done New false 3295 Acked 4294 Write 9 Done Done New false 3296 Acked 4295 Read 8 Done Done New false 3297 Acked 4296 Write 7 Done Done New false 3298 Acked 4297 Write 8 Done Done New false 3299 Acked 4298 Flush 0 Done Done New false 3300 Acked 4299 Read 7 Done Done New false 3301 Acked 4300 Read 4 Done Done New false 3302 Acked 4301 Write 4 Done Done New false 3303 Acked 4302 Read 10 Done Done New false 3304 Acked 4303 Write 4 Done Done New false 3305 Acked 4304 Read 9 Done Done New false 3306 Acked 4305 Write 3 Done Done New false 3307 Acked 4306 Read 9 Done Done New false 3308 Acked 4307 Write 3 Done Done New false 3309 Acked 4308 Read 7 Done Done New false 3310 Acked 4309 Write 8 Done Done New false 3311 Acked 4310 Write 1 Done Done New false 3312 Acked 4311 Write 5 Done Done New false 3313 Acked 4312 Write 6 Done Done New false 3314 Acked 4313 Read 5 Done Done New false 3315 Acked 4314 Write 5 Done Done New false 3316 Acked 4315 Read 10 Done Done New false 3317 Acked 4316 Write 6 Done Done New false 3318 Acked 4317 Read 9 Done Done New false 3319 Acked 4318 Read 10 Done Done New false 3320 Acked 4319 Read 9 Done Done New false 3321 Acked 4320 Read 8 Done Done New false 3322 Acked 4321 Read 1 Done Done New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 301 301 Sent 0 0 0 0 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4298 4298 4020 Downstairs last five completed: 4020 4019 4018 4017 4016 Upstairs last five completed: 3322 3321 3320 3319 3318 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:44.904758335Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:44.904825877Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:44.904854239Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:44.904880779Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:44.904901681Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:44.904942744Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:44.905179747Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:45.906315869Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:45.906427023Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:45.906503349Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:45.906566737Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:45.906591401Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:45.906631162Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:45.906856103Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1196) [8830][2] reports Running {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.908010802Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.908182965Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910121036Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910156517Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910180247Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 4020","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910462726Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910496527Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910519553Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910548757Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:46.910571808Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4322 4321 4320 4319 4318 Upstairs last five completed: 3323 3322 3321 3320 3319 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:32:48.512 INFO request completed, latency_us: 464, response_code: 204, uri: /stop/cid/0, method: POST[8810] Got stop action so:false kr:false , req_id: 9a6fe4a1-ae1a-40e2-b5cc-359672147390, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:32:48.514 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/0, method: GET, req_id: 44298969-c560-4e06-8cc8-cb7a66daba66, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:48.51431822Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:48.514455134Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:48.514510755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:48.514539972Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:48.514563127Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:48.514807835Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:49.515843336Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:49.515914348Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:49.515942817Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:49.515979998Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:49.516001519Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:49.516024311Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:49.516072114Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:50.517299298Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:50.517367932Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:50.517396116Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:50.517422542Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:50.517442916Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:50.517467563Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:50.51751305Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:51.518764977Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:51.518843215Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:51.518870887Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:51.518904477Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:51.518931189Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:51.518951515Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:51.518973771Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:51.519078599Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:32:52.516 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/0, method: GET, req_id: dab82b21-236c-4d58-892d-811559a693b1, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 3440 len 32768 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:52.520719848Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:52.520807895Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:52.52083867Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:52.520866067Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:52.520886468Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 1 jobs since flush 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:52.520915546Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:52.52100896Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 002/300 Read block 271 len 8192 003/300 Read block 4906 len 16384 004/300 Read block 2909 len 12288 005/300 Write block 2154 len 24576 data: 3 4 4 4 5 4 006/300 Read block 475 len 32768 007/300 Write block 3924 len 12288 data: 5 5 6 008/300 Flush 009/300 Flush 010/300 Write block 4258 len 12288 data: 4 4 5 011/300 Write block 2002 len 4096 data: 4 012/300 Read block 2187 len 24576 013/300 Read block 567 len 8192 014/300 Read block 3707 len 8192 015/300 Read block 1820 len 28672 016/300 Write block 3326 len 28672 data: 3 2 2 2 2 2 5 017/300 Read block 235 len 24576 018/300 Write block 2503 len 32768 data: 3 3 4 5 5 6 5 6 019/300 Write block 2920 len 28672 data: 4 4 4 3 3 2 2 020/300 Flush 021/300 Write block 3306 len 8192 data: 3 2 022/300 Write block 4012 len 32768 data: 4 4 5 4 4 4 4 3 023/300 Flush 024/300 Read block 1119 len 16384 025/300 Read block 1857 len 4096 026/300 Read block 4396 len 24576 027/300 Read block 157 len 20480 028/300 Flush 029/300 Read block 2279 len 20480 030/300 Write block 2336 len 24576 data: 4 5 4 3 4 2 031/300 Read block 3039 len 8192 032/300 Write block 1823 len 40960 data: 3 3 3 3 3 2 2 2 3 3 033/300 Flush 034/300 Write block 2615 len 20480 data: 4 3 2 2 2 035/300 Read block 4810 len 12288 036/300 Read block 1957 len 4096 037/300 Read block 85 len 36864 038/300 Read block 2186 len 8192 039/300 Write block 2454 len 4096 data: 3 040/300 Read block 2802 len 28672 041/300 Write block 3154 len 24576 data: 5 4 3 3 2 2 042/300 Read block 707 len 8192 043/300 Write block 1133 len 28672 data: 4 6 6 6 6 5 3 044/300 Write block 3109 len 20480 data: 3 2 2 2 3 045/300 Flush 046/300 Read block 4757 len 16384 047/300 Write block 3977 len 36864 data: 3 3 3 3 3 2 2 3 3 048/300 Read block 3736 len 8192 049/300 Read block 798 len 20480 050/300 Write block 4565 len 4096 data: 3 051/300 Read block 4542 len 16384 052/300 Read block 2351 len 40960 053/300 Write block 4900 len 36864 data: 3 3 4 5 5 4 4 5 4 054/300 Write block 800 len 16384 data: 2 4 4 4 055/300 Read block 305 len 36864 056/300 Read block 42 len 36864 057/300 Read block 750 len 28672 058/300 Write block 4803 len 24576 data: 5 5 6 5 4 3 059/300 Write block 4419 len 20480 data: 4 4 4 5 5 060/300 Read block 3087 len 28672 061/300 Read block 3192 len 36864 062/300 Read block 4277 len 24576 063/300 Flush 064/300 Read block 1255 len 16384 065/300 Read block 789 len 32768 066/300 Read block 782 len 40960 067/300 Flush 068/300 Read block 4088 len 16384 069/300 Write block 4626 len 4096 data: 5 070/300 Read block 1516 len 4096 071/300 Read block 2141 len 24576 072/300 Read block 428 len 16384 073/300 Read block 1953 len 12288 074/300 Write block 388 len 8192 data: 2 2 075/300 Write block 3910 len 4096 data: 4 076/300 Read block 2741 len 40960 077/300 Read block 1200 len 4096 078/300 Read block 922 len 40960 079/300 Read block 4139 len 20480 080/300 Read block 4435 len 16384 081/300 Read block 3746 len 36864 082/300 Write block 4981 len 28672 data: 4 4 6 5 4 5 4 083/300 Flush 084/300 Flush 085/300 Read block 1504 len 20480 086/300 Write block 4213 len 24576 data: 2 3 4 4 4 4 087/300 Write block 1704 len 12288 data: 2 3 3 088/300 Read block 2606 len 20480 089/300 Read block 2400 len 16384 090/300 Flush 091/300 Flush 092/300 Read block 2383 len 16384 093/300 Read block 1296 len 28672 094/300 Read block 1627 len 32768 095/300 Write block 4550 len 8192 data: 2 3 096/300 Read block 3496 len 16384 097/300 Read block 1723 len 20480 098/300 Write block 799 len 4096 data: 2 099/300 Write block 1556 len 24576 data: 4 4 5 4 4 4 100/300 Read block 3774 len 16384 101/300 Read block 3028 len 24576 102/300 Write block 2158 len 36864 data: 6 5 5 4 4 3 5 5 4 103/300 Read block 3706 len 8192 104/300 Read block 3052 len 8192 105/300 Flush 106/300 Write block 1963 len 8192 data: 3 3 107/300 Write block 2511 len 32768 data: 6 4 4 3 2 2 2 2 108/300 Read block 3028 len 20480 109/300 Flush 110/300 Read block 793 len 36864 111/300 Read block 4595 len 40960 112/300 Write block 3178 len 36864 data: 4 4 4 3 3 3 3 2 3 113/300 Read block 2979 len 4096 114/300 Write block 2328 len 8192 data: 4 5 115/300 Read block 596 len 40960 116/300 Read block 833 len 32768 117/300 Write block 4076 len 36864 data: 3 2 2 2 3 3 4 4 4 118/300 Write block 834 len 8192 data: 2 3 119/300 Write block 1998 len 20480 data: 2 4 4 5 5 120/300 Flush 121/300 Flush 122/300 Read block 325 len 24576 123/300 Read block 3413 len 32768 124/300 Write block 3736 len 20480 data: 3 3 3 4 4 125/300 Read block 1775 len 4096 126/300 Read block 4551 len 16384 127/300 Read block 2870 len 40960 128/300 Read block 2752 len 4096 129/300 Read block 4482 len 32768 130/300 Read block 890 len 8192 131/300 Read block 3135 len 24576 132/300 Write block 2322 len 32768 data: 2 2 3 3 4 4 5 6 133/300 Write block 2639 len 16384 data: 3 3 3 3 134/300 Read block 52 len 40960 135/300 Read block 4879 len 12288 136/300 Write block 4274 len 32768 data: 3 2 2 2 3 3 3 3 137/300 Read block 1897 len 24576 138/300 Read block 1080 len 4096 139/300 Read block 3277 len 12288 140/300 Write block 1853 len 16384 data: 2 2 2 2 141/300 Write block 2986 len 32768 data: 4 4 3 4 4 4 3 3 142/300 Write block 479 len 28672 data: 3 2 2 2 2 2 2 143/300 Write block 316 len 32768 data: 5 4 5 6 4 5 5 5 144/300 Read block 1879 len 16384 145/300 Write block 3713 len 32768 data: 2 3 2 3 3 2 2 2 146/300 Read block 2515 len 24576 147/300 Flush 148/300 Read block 484 len 32768 149/300 Flush 150/300 Write block 4631 len 12288 data: 5 5 6 151/300 Flush 152/300 Read block 4276 len 32768 153/300 Flush 154/300 Read block 4458 len 36864 155/300 Read block 2919 len 36864 156/300 Write block 2245 len 20480 data: 2 2 2 2 2 157/300 Write block 2705 len 8192 data: 3 3 158/300 Read block 1656 len 32768 159/300 Flush 160/300 Write block 248 len 36864 data: 3 2 2 2 2 2 3 3 4 161/300 Read block 4805 len 28672 162/300 Write block 80 len 28672 data: 3 3 3 4 6 6 6 163/300 Read block 2409 len 36864 164/300 Write block 1450 len 8192 data: 3 2 165/300 Flush 166/300 Read block 4919 len 36864 167/300 Write block 1551 len 16384 data: 2 2 2 2 168/300 Write block 1832 len 12288 data: 4 2 2 169/300 Read block 1687 len 12288 170/300 Write block 167 len 24576 data: 3 3 3 3 3 3 171/300 Write block 470 len 24576 data: 2 2 2 2 3 3 172/300 Read block 2370 len 36864 173/300 Read block 1401 len 40960 174/300 Read block 4161 len 36864 175/300 Write block 3633 len 28672 data: 4 4 3 3 4 4 3 176/300 Read block 1591 len 8192 177/300 Flush 178/300 Read block 438 len 28672 179/300 Write block 412 len 40960 data: 4 3 2 2 4 4 5 5 5 5 180/300 Write block 3351 len 40960 data: 4 3 2 3 3 3 3 4 6 6 181/300 Flush 182/300 Read block 305 len 12288 183/300 Read block 2917 len 12288 184/300 Read block 2956 len 32768 185/300 Read block 1784 len 24576 186/300 Flush 187/300 Read block 4962 len 24576 188/300 Write block 2984 len 4096 data: 3 189/300 Read block 2898 len 8192 190/300 Write block 3984 len 40960 data: 4 4 3 4 4 4 5 5 5 7 191/300 Write block 4575 len 24576 data: 4 4 5 5 6 6 192/300 Flush 193/300 Read block 568 len 12288 194/300 Read block 3959 len 24576 195/300 Read block 4385 len 12288 196/300 Write block 2509 len 40960 data: 6 7 7 5 5 4 3 3 3 3 197/300 Write block 2668 len 4096 data: 2 198/300 Write block 1322 len 12288 data: 3 3 3 199/300 Flush 200/300 Write block 3247 len 24576 data: 3 3 3 2 3 4 201/300 Write block 3511 len 12288 data: 2 2 3 202/300 Write block 4151 len 28672 data: 3 2 2 2 3 3 3 203/300 Write block 3606 len 24576 data: 4 4 4 5 5 5 204/300 Write block 2816 len 32768 data: 3 2 2 3 4 6 6 6 205/300 Read block 1283 len 40960 206/300 Write block 834 len 8192 data: 3 4 207/300 Flush 208/300 Read block 4111 len 24576 209/300 Read block 1327 len 40960 210/300 Read block 3973 len 8192 211/300 Write block 2192 len 24576 data: 3 2 2 2 5 4 212/300 Write block 4759 len 32768 data: 4 3 4 4 3 3 3 4 213/300 Write block 4200 len 24576 data: 2 2 3 3 4 4 214/300 Write block 3319 len 32768 data: 3 3 2 2 2 3 3 4 215/300 Read block 31 len 32768 216/300 Read block 2170 len 20480 217/300 Flush 218/300 Read block 1438 len 8192 219/300 Read block 4669 len 40960 220/300 Write block 2475 len 12288 data: 3 3 3 221/300 Write block 4521 len 20480 data: 4 4 2 2 2 222/300 Read block 149 len 12288 223/300 Flush 224/300 Write block 4268 len 8192 data: 7 7 225/300 Read block 4663 len 20480 226/300 Flush 227/300 Read block 193 len 12288 228/300 Read block 612 len 24576 229/300 Write block 436 len 24576 data: 5 8 9 9 9 7 230/300 Read block 479 len 4096 231/300 Read block 2751 len 36864 232/300 Write block 3353 len 12288 data: 3 4 4 233/300 Read block 4429 len 12288 234/300 Write block 4885 len 16384 data: 6 5 6 4 235/300 Write block 3985 len 40960 data: 5 4 5 5 5 6 6 6 8 7 236/300 Write block 465 len 24576 data: 3 2 2 2 2 3 237/300 Read block 2423 len 16384 238/300 Read block 4247 len 24576 239/300 Flush 240/300 Write block 1185 len 24576 data: 2 2 3 3 3 3 241/300 Write block 3673 len 12288 data: 3 2 2 242/300 Write block 3839 len 24576 data: 3 3 4 3 3 3 243/300 Flush 244/300 Read block 4081 len 28672 245/300 Read block 1263 len 20480 246/300 Write block 811 len 12288 data: 4 5 5 247/300 Write block 4145 len 24576 data: 7 6 5 5 4 3 248/300 Read block 1639 len 28672 249/300 Write block 844 len 12288 data: 4 4 5 250/300 Read block 4557 len 40960 251/300 Write block 4290 len 28672 data: 2 2 2 3 3 3 3 252/300 Write block 412 len 12288 data: 5 4 3 253/300 Flush 254/300 Read block 1002 len 28672 255/300 Write block 3318 len 16384 data: 3 4 4 3 256/300 Read block 901 len 36864 257/300 Write block 744 len 16384 data: 3 3 2 2 258/300 Write block 3470 len 40960 data: 4 3 3 3 4 3 3 3 3 3 259/300 Read block 2355 len 36864 260/300 Read block 3611 len 16384 261/300 Write block 3601 len 24576 data: 3 5 5 4 4 5 262/300 Read block 2532 len 28672 263/300 Read block 333 len 24576 264/300 Write block 4256 len 24576 data: 5 5 5 5 6 5 265/300 Read block 2834 len 40960 266/300 Read block 832 len 36864 267/300 Write block 1494 len 8192 data: 4 5 268/300 Write block 144 len 12288 data: 3 3 4 269/300 Read block 4649 len 12288 270/300 Write block 3267 len 12288 data: 4 4 3 271/300 Read block 4262 len 8192 272/300 Read block 4390 len 32768 273/300 Write block 2618 len 16384 data: 3 3 3 3 274/300 Write block 4878 len 4096 data: 5 275/300 Write block 21 len 36864 data: 3 3 4 4 5 4 5 5 4 276/300 Read block 2747 len 40960 277/300 Read block 560 len 36864 278/300 Read block 1496 len 28672 279/300 Read block 1084 len 40960 280/300 Read block 3155 len 36864 281/300 Read block 2463 len 24576 282/300 Read block 4298 len 4096 283/300 Read block 1527 len 12288 284/300 Write block 176 len 8192 data: 2 2 285/300 Read block 95 len 4096 286/300 Read block 1592 len 24576 287/300 Write block 1394 len 16384 data: 4 4 4 2 288/300 Read block 1353 len 16384 289/300 Write block 4655 len 16384 data: 5 5 4 4 290/300 Read block 3496 len 36864 291/300 Read block 1666 len 16384 292/300 Write block 429 len 40960 data: 4 4 5 4 3 3 3 6 9 10 293/300 Read block 3363 len 32768 294/300 Read block 1264 len 40960 295/300 Read block 1565 len 28672 296/300 Read block 2834 len 24576 297/300 Read block 787 len 12288 298/300 Read block 4063 len 36864 299/300 Write block 3046 len 12288 data: 3 4 6 300/300 Read block 4593 len 16384 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:32:53.135 INFO request completed, latency_us: 305, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7aab2262-3511-4f63-b186-0e2e014907cd, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 3324 Acked 4323 Read 8 New Done Done true 3325 Acked 4324 Read 2 New Done Done false 3326 Acked 4325 Read 4 New Done Done false 3327 Acked 4326 Read 3 New Done Done false 3328 Acked 4327 Write 6 New Done Done false 3329 Acked 4328 Read 8 New Done Done false 3330 Acked 4329 Write 3 New Done Done false 3331 Acked 4330 Flush 0 New Done Done false 3332 Acked 4331 Flush 0 New Done Done false 3333 Acked 4332 Write 3 New Done Done false 3334 Acked 4333 Write 1 New Done Done false 3335 Acked 4334 Read 6 New Done Done false 3336 Acked 4335 Read 2 New Done Done false 3337 Acked 4336 Read 2 New Done Done false 3338 Acked 4337 Read 7 New Done Done false 3339 Acked 4338 Write 7 New Done Done false 3340 Acked 4339 Read 6 New Done Done false 3341 Acked 4340 Write 8 New Done Done false 3342 Acked 4341 Write 7 New Done Done false 3343 Acked 4342 Flush 0 New Done Done false 3344 Acked 4343 Write 2 New Done Done false 3345 Acked 4344 Write 8 New Done Done false 3346 Acked 4345 Flush 0 New Done Done false 3347 Acked 4346 Read 4 New Done Done false 3348 Acked 4347 Read 1 New Done Done false 3349 Acked 4348 Read 6 New Done Done false 3350 Acked 4349 Read 5 New Done Done false 3351 Acked 4350 Flush 0 New Done Done false 3352 Acked 4351 Read 5 New Done Done false 3353 Acked 4352 Write 6 New Done Done false 3354 Acked 4353 Flush 0 New Done Done false 3355 Acked 4354 Read 2 New Done Done false 3356 Acked 4355 Write 10 New Done Done false 3357 Acked 4356 Flush 0 New Done Done false 3358 Acked 4357 Write 5 New Done Done false 3359 Acked 4358 Read 3 New Done Done false 3360 Acked 4359 Read 1 New Done Done false 3361 Acked 4360 Read 9 New Done Done false 3362 Acked 4361 Read 2 New Done Done false 3363 Acked 4362 Write 1 New Done Done false 3364 Acked 4363 Read 7 New Done Done false 3365 Acked 4364 Write 6 New Done Done false 3366 Acked 4365 Read 2 New Done Done false 3367 Acked 4366 Write 7 New Done Done false 3368 Acked 4367 Write 5 New Done Done false 3369 Acked 4368 Flush 0 New Done Done false 3370 Acked 4369 Read 4 New Done Done false 3371 Acked 4370 Write 9 New Done Done false 3372 Acked 4371 Read 2 New Done Done false 3373 Acked 4372 Read 5 New Done Done false 3374 Acked 4373 Write 1 New Done Done false 3375 Acked 4374 Read 4 New Done Done false 3376 Acked 4375 Read 10 New Done Done false 3377 Acked 4376 Write 9 New Done Done false 3378 Acked 4377 Write 4 New Done Done false 3379 Acked 4378 Read 9 New Done Done false 3380 Acked 4379 Read 9 New Done Done false 3381 Acked 4380 Read 7 New Done Done false 3382 Acked 4381 Write 6 New Done Done false 3383 Acked 4382 Write 5 New Done Done false 3384 Acked 4383 Read 7 New Done Done false 3385 Acked 4384 Read 9 New Done Done false 3386 Acked 4385 Read 6 New Done Done false 3387 Acked 4386 Flush 0 New Done Done false 3388 Acked 4387 Read 4 New Done Done false 3389 Acked 4388 Read 8 New Done Done false 3390 Acked 4389 Read 10 New Done Done false 3391 Acked 4390 Flush 0 New Done Done false 3392 Acked 4391 Read 4 New Done Done false 3393 Acked 4392 Write 1 New Done Done false 3394 Acked 4393 Read 1 New Done Done false 3395 Acked 4394 Read 6 New Done Done false 3396 Acked 4395 Read 4 New Done Done false 3397 Acked 4396 Read 3 New Done Done false 3398 Acked 4397 Write 2 New Done Done false 3399 Acked 4398 Write 1 New Done Done false 3400 Acked 4399 Read 10 New Done Done false 3401 Acked 4400 Read 1 New Done Done false 3402 Acked 4401 Read 10 New Done Done false 3403 Acked 4402 Read 5 New Done Done false 3404 Acked 4403 Read 4 New Done Done false 3405 Acked 4404 Read 9 New Done Done false 3406 Acked 4405 Write 7 New Done Done false 3407 Acked 4406 Flush 0 New Done Done false 3408 Acked 4407 Flush 0 New Done Done false 3409 Acked 4408 Read 5 New Done Done false 3410 Acked 4409 Write 6 New Done Done false 3411 Acked 4410 Write 3 New Done Done false 3412 Acked 4411 Read 5 New Done Done false 3413 Acked 4412 Read 4 New Done Done false 3414 Acked 4413 Flush 0 New Done Done false 3415 Acked 4414 Flush 0 New Done Done false 3416 Acked 4415 Read 4 New Done Done false 3417 Acked 4416 Read 7 New Done Done false 3418 Acked 4417 Read 8 New Done Done false 3419 Acked 4418 Write 2 New Done Done false 3420 Acked 4419 Read 4 New Done Done false 3421 Acked 4420 Read 5 New Done Done false 3422 Acked 4421 Write 1 New Done Done false 3423 Acked 4422 Write 6 New Done Done false 3424 Acked 4423 Read 4 New Done Done false 3425 Acked 4424 Read 6 New Done Done false 3426 Acked 4425 Write 9 New Done Done false 3427 Acked 4426 Read 2 New Done Done false 3428 Acked 4427 Read 2 New Done Done false 3429 Acked 4428 Flush 0 New Done Done false 3430 Acked 4429 Write 2 New Done Done false 3431 Acked 4430 Write 8 New Done Done false 3432 Acked 4431 Read 5 New Done Done false 3433 Acked 4432 Flush 0 New Done Done false 3434 Acked 4433 Read 9 New Done Done false 3435 Acked 4434 Read 10 New Done Done false 3436 Acked 4435 Write 9 New Done Done false 3437 Acked 4436 Read 1 New Done Done false 3438 Acked 4437 Write 2 New Done Done false 3439 Acked 4438 Read 10 New Done Done false 3440 Acked 4439 Read 8 New Done Done false 3441 Acked 4440 Write 9 New Done Done false 3442 Acked 4441 Write 2 New Done Done false 3443 Acked 4442 Write 5 New Done Done false 3444 Acked 4443 Flush 0 New Done Done false 3445 Acked 4444 Flush 0 New Done Done false 3446 Acked 4445 Read 6 New Done Done false 3447 Acked 4446 Read 8 New Done Done false 3448 Acked 4447 Write 5 New Done Done false 3449 Acked 4448 Read 1 New Done Done false 3450 Acked 4449 Read 4 New Done Done false 3451 Acked 4450 Read 10 New Done Done false 3452 Acked 4451 Read 1 New Done Done false 3453 Acked 4452 Read 8 New Done Done false 3454 Acked 4453 Read 2 New Done Done false 3455 Acked 4454 Read 6 New Done Done false 3456 Acked 4455 Write 8 New Done Done false 3457 Acked 4456 Write 4 New Done Done false 3458 Acked 4457 Read 10 New Done Done false 3459 Acked 4458 Read 3 New Done Done false 3460 Acked 4459 Write 8 New Done Done false 3461 Acked 4460 Read 6 New Done Done false 3462 Acked 4461 Read 1 New Done Done false 3463 Acked 4462 Read 3 New Done Done false 3464 Acked 4463 Write 4 New Done Done false 3465 Acked 4464 Write 8 New Done Done false 3466 Acked 4465 Write 7 New Done Done false 3467 Acked 4466 Write 8 New Done Done false 3468 Acked 4467 Read 4 New Done Done false 3469 Acked 4468 Write 8 New Done Done false 3470 Acked 4469 Read 6 New Done Done false 3471 Acked 4470 Flush 0 New Done Done false 3472 Acked 4471 Read 8 New Done Done false 3473 Acked 4472 Flush 0 New Done Done false 3474 Acked 4473 Write 3 New Done Done false 3475 Acked 4474 Flush 0 New Done Done false 3476 Acked 4475 Read 8 New Done Done false 3477 Acked 4476 Flush 0 New Done Done false 3478 Acked 4477 Read 9 New Done Done false 3479 Acked 4478 Read 9 New Done Done false 3480 Acked 4479 Write 5 New Done Done false 3481 Acked 4480 Write 2 New Done Done false 3482 Acked 4481 Read 8 New Done Done false 3483 Acked 4482 Flush 0 New Done Done false 3484 Acked 4483 Write 9 New Done Done false 3485 Acked 4484 Read 7 New Done Done false 3486 Acked 4485 Write 7 New Done Done false 3487 Acked 4486 Read 9 New Done Done false 3488 Acked 4487 Write 2 New Done Done false 3489 Acked 4488 Flush 0 New Done Done false 3490 Acked 4489 Read 9 New Done Done false 3491 Acked 4490 Write 4 New Done Done false 3492 Acked 4491 Write 3 New Done Done false 3493 Acked 4492 Read 3 New Done Done false 3494 Acked 4493 Write 6 New Done Done false 3495 Acked 4494 Write 6 New Done Done false 3496 Acked 4495 Read 9 New Done Done false 3497 Acked 4496 Read 10 New Done Done false 3498 Acked 4497 Read 9 New Done Done false 3499 Acked 4498 Write 7 New Done Done false 3500 Acked 4499 Read 2 New Done Done false 3501 Acked 4500 Flush 0 New Done Done false 3502 Acked 4501 Read 7 New Done Done false 3503 Acked 4502 Write 10 New Done Done false 3504 Acked 4503 Write 10 New Done Done false 3505 Acked 4504 Flush 0 New Done Done false 3506 Acked 4505 Read 3 New Done Done false 3507 Acked 4506 Read 3 New Done Done false 3508 Acked 4507 Read 8 New Done Done false 3509 Acked 4508 Read 6 New Done Done false 3510 Acked 4509 Flush 0 New Done Done false 3511 Acked 4510 Read 6 New Done Done false 3512 Acked 4511 Write 1 New Done Done false 3513 Acked 4512 Read 2 New Done Done false 3514 Acked 4513 Write 10 New Done Done false 3515 Acked 4514 Write 6 New Done Done false 3516 Acked 4515 Flush 0 New Done Done false 3517 Acked 4516 Read 3 New Done Done false 3518 Acked 4517 Read 6 New Done Done false 3519 Acked 4518 Read 3 New Done Done false 3520 Acked 4519 Write 10 New Done Done false 3521 Acked 4520 Write 1 New Done Done false 3522 Acked 4521 Write 3 New Done Done false 3523 Acked 4522 Flush 0 New Done Done false 3524 Acked 4523 Write 6 New Done Done false 3525 Acked 4524 Write 3 New Done Done false 3526 Acked 4525 Write 7 New Done Done false 3527 Acked 4526 Write 6 New Done Done false 3528 Acked 4527 Write 8 New Done Done false 3529 Acked 4528 Read 10 New Done Done false 3530 Acked 4529 Write 2 New Done Done false 3531 Acked 4530 Flush 0 New Done Done false 3532 Acked 4531 Read 6 New Done Done false 3533 Acked 4532 Read 10 New Done Done false 3534 Acked 4533 Read 2 New Done Done false 3535 Acked 4534 Write 6 New Done Done false 3536 Acked 4535 Write 8 New Done Done false 3537 Acked 4536 Write 6 New Done Done false 3538 Acked 4537 Write 8 New Done Done false 3539 Acked 4538 Read 8 New Done Done false 3540 Acked 4539 Read 5 New Done Done false 3541 Acked 4540 Flush 0 New Done Done false 3542 Acked 4541 Read 2 New Done Done false 3543 Acked 4542 Read 10 New Done Done false 3544 Acked 4543 Write 3 New Done Done false 3545 Acked 4544 Write 5 New Done Done false 3546 Acked 4545 Read 3 New Done Done false 3547 Acked 4546 Flush 0 New Done Done false 3548 Acked 4547 Write 2 New Done Done false 3549 Acked 4548 Read 5 New Done Done false 3550 Acked 4549 Flush 0 New Done Done false 3551 Acked 4550 Read 3 New Done Done false 3552 Acked 4551 Read 6 New Done Done false 3553 Acked 4552 Write 6 New Done Done false 3554 Acked 4553 Read 1 New Done Done false 3555 Acked 4554 Read 9 New Done Done false 3556 Acked 4555 Write 3 New Done Done false 3557 Acked 4556 Read 3 New Done Done false 3558 Acked 4557 Write 4 New Done Done false 3559 Acked 4558 Write 10 New Done Done false 3560 Acked 4559 Write 6 New Done Done false 3561 Acked 4560 Read 4 New Done Done false 3562 Acked 4561 Read 6 New Done Done false 3563 Acked 4562 Flush 0 New Done Done false 3564 Acked 4563 Write 6 New Done Done false 3565 Acked 4564 Write 3 New Done Done false 3566 Acked 4565 Write 6 New Done Done false 3567 Acked 4566 Flush 0 New Done Done false 3568 Acked 4567 Read 7 New Done Done false 3569 Acked 4568 Read 5 New Done Done false 3570 Acked 4569 Write 3 New Done Done false 3571 Acked 4570 Write 6 New Done Done false 3572 Acked 4571 Read 7 New Done Done false 3573 Acked 4572 Write 3 New Done Done false 3574 Acked 4573 Read 10 New Done Done false 3575 Acked 4574 Write 7 New Done Done false 3576 Acked 4575 Write 3 New Done Done false 3577 Acked 4576 Flush 0 New Done Done false 3578 Acked 4577 Read 7 New Done Done false 3579 Acked 4578 Write 4 New Done Done false 3580 Acked 4579 Read 9 New Done Done false 3581 Acked 4580 Write 4 New Done Done false 3582 Acked 4581 Write 10 New Done Done false 3583 Acked 4582 Read 9 New Done Done false 3584 Acked 4583 Read 4 New Done Done false 3585 Acked 4584 Write 6 New Done Done false 3586 Acked 4585 Read 7 New Done Done false 3587 Acked 4586 Read 6 New Done Done false 3588 Acked 4587 Write 6 New Done Done false 3589 Acked 4588 Read 10 New Done Done false 3590 Acked 4589 Read 9 New Done Done false 3591 Acked 4590 Write 2 New Done Done false 3592 Acked 4591 Write 3 New Done Done false 3593 Acked 4592 Read 3 New Done Done false 3594 Acked 4593 Write 3 New Done Done false 3595 Acked 4594 Read 2 New Done Done false 3596 Acked 4595 Read 8 New Done Done false 3597 Acked 4596 Write 4 New Done Done false 3598 Acked 4597 Write 1 New Done Done false 3599 Acked 4598 Write 9 New Done Done false 3600 Acked 4599 Read 10 New Done Done false 3601 Acked 4600 Read 9 New Done Done false 3602 Acked 4601 Flush 0 New Done Done false 3603 Acked 4602 Read 7 New Done Done false 3604 Acked 4603 Read 10 New Done Done false 3605 Acked 4604 Read 9 New Done Done false 3606 Acked 4605 Read 6 New Done Done false 3607 Acked 4606 Read 1 New Done Done false 3608 Acked 4607 Read 3 New Done Done false 3609 Acked 4608 Write 2 New Done Done false 3610 Acked 4609 Read 1 New Done Done false 3611 Acked 4610 Read 6 New Done Done false 3612 Acked 4611 Write 4 New Done Done false 3613 Acked 4612 Read 4 New Done Done false 3614 Acked 4613 Write 4 New Done Done false 3615 Acked 4614 Read 9 New Done Done false 3616 Acked 4615 Read 4 New Done Done false 3617 Acked 4616 Write 10 New Done Done false 3618 Acked 4617 Read 8 New Done Done false 3619 Acked 4618 Read 10 New Done Done false 3620 Acked 4619 Read 7 New Done Done false 3621 Acked 4620 Read 6 New Done Done false 3622 Acked 4621 Read 3 New Done Done false 3623 Acked 4622 Read 9 New Done Done false 3624 Acked 4623 Write 3 New Done Done false 3625 Acked 4624 Read 4 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4322 4601 4601 Downstairs last five completed: 4322 4321 4320 4319 4318 Upstairs last five completed: 3625 3624 3623 3622 3621 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:53.522614607Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:53.522695746Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:53.522725813Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:53.522752301Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:53.522772728Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:53.522803443Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:53.523045539Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:54.523373906Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:54.523490474Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:54.523553797Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:54.523580704Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:54.523601303Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 303 jobs since flush 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:54.523624031Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:54.523937946Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1197) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.524974741Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.525137644Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.52702647Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.527058017Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.527081743Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 4322","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.527366794Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.527396625Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.527419875Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.527446955Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:55.527469834Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4625 4624 4623 4622 4621 Upstairs last five completed: 3626 3625 3624 3623 3622 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(2) stop 2 [8830] Got stop action so:false kr:false Jan 02 22:32:57.138 INFO request completed, latency_us: 459, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 51b8b0d4-eb82-49af-9844-bf2c3d741950, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:32:57.140 INFO request completed, latency_us: 354, response_code: 200, uri: /state/cid/2, method: GET, req_id: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:57.140071739Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 3a3e29f7-a1ad-4f21-942a-f0b74534bea4, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:57.140199779Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 9998 {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:57.140285684Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:57.140326342Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:57.140363405Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:57.140478526Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:58.142024768Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:58.142090761Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:58.142119208Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:58.142145735Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:58.142166072Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:58.142197711Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:58.142259032Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:59.143567517Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:59.143644013Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:59.143681156Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:32:59.143708771Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:59.143729368Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:59.143751765Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:32:59.143782381Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:00.144915834Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:00.145016714Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:00.145044824Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:00.145071389Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:00.145091833Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:00.145132892Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:00.145194009Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:33:01.142 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/2, method: GET, req_id: 83ee6129-2a9e-4148-aec6-e1f582a90d2d, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 2372 len 12288 data: 2 4 3 002/300 Read block 286 len 32768 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:01.146430087Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:01.146495395Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:01.146523781Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:01.147278549Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:01.147321433Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 2 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:01.147359315Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:01.147636405Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 3842 len 16384 004/300 Write block 4740 len 16384 data: 6 5 3 2 005/300 Write block 4133 len 12288 data: 4 4 3 006/300 Write block 942 len 20480 data: 4 4 3 4 5 007/300 Write block 2277 len 4096 data: 5 008/300 Read block 4521 len 12288 009/300 Write block 271 len 24576 data: 7 7 6 5 6 5 010/300 Write block 2607 len 32768 data: 3 4 4 4 2 3 4 4 011/300 Write block 4885 len 16384 data: 7 6 7 5 012/300 Write block 1021 len 40960 data: 4 4 4 3 3 3 2 2 2 2 013/300 Write block 4215 len 4096 data: 5 014/300 Read block 771 len 16384 015/300 Write block 4213 len 40960 data: 3 4 6 5 5 5 4 4 3 3 016/300 Read block 141 len 12288 017/300 Read block 3614 len 36864 018/300 Write block 1035 len 16384 data: 2 3 3 3 019/300 Write block 3774 len 28672 data: 3 3 3 4 5 5 4 020/300 Write block 1755 len 24576 data: 5 5 5 5 5 4 021/300 Write block 1639 len 36864 data: 3 3 3 2 2 3 3 2 3 022/300 Write block 1321 len 32768 data: 2 4 4 4 4 4 5 4 023/300 Write block 3073 len 12288 data: 2 3 3 024/300 Read block 4383 len 36864 025/300 Read block 967 len 8192 026/300 Read block 62 len 40960 027/300 Write block 4407 len 16384 data: 2 2 3 3 028/300 Read block 1192 len 16384 029/300 Flush 030/300 Read block 280 len 12288 031/300 Write block 4488 len 36864 data: 2 3 3 3 3 3 3 2 2 032/300 Write block 2633 len 24576 data: 4 4 4 3 3 3 033/300 Read block 3588 len 32768 034/300 Write block 3539 len 36864 data: 5 5 6 6 6 5 4 2 2 035/300 Read block 821 len 32768 036/300 Write block 2371 len 40960 data: 2 3 5 4 3 3 3 2 2 2 037/300 Write block 2953 len 32768 data: 6 5 4 4 2 2 2 2 038/300 Read block 509 len 40960 039/300 Write block 2301 len 40960 data: 4 4 4 5 4 4 4 4 3 3 040/300 Read block 2831 len 4096 041/300 Read block 3854 len 36864 042/300 Write block 1414 len 8192 data: 2 2 043/300 Write block 3414 len 32768 data: 3 3 4 5 5 4 4 4 044/300 Read block 2389 len 8192 045/300 Read block 893 len 32768 046/300 Write block 2956 len 16384 data: 5 3 3 3 047/300 Read block 1617 len 20480 048/300 Read block 2027 len 20480 049/300 Flush 050/300 Read block 4399 len 40960 051/300 Write block 4383 len 4096 data: 4 052/300 Read block 93 len 40960 053/300 Write block 2327 len 36864 data: 5 6 7 5 5 5 3 5 4 054/300 Write block 4573 len 16384 data: 2 3 5 5 055/300 Read block 2763 len 40960 056/300 Read block 1101 len 24576 057/300 Read block 4253 len 16384 058/300 Read block 1116 len 28672 059/300 Read block 2065 len 24576 060/300 Read block 1631 len 28672 061/300 Write block 64 len 16384 data: 3 2 2 2 062/300 Write block 4258 len 24576 data: 6 6 7 6 5 4 063/300 Write block 4952 len 36864 data: 3 3 3 3 3 5 5 3 3 064/300 Read block 4351 len 40960 065/300 Write block 1048 len 28672 data: 2 2 3 3 3 3 3 066/300 Write block 4830 len 4096 data: 3 067/300 Read block 2263 len 32768 068/300 Write block 1558 len 8192 data: 6 5 069/300 Read block 4470 len 4096 070/300 Flush 071/300 Read block 482 len 24576 072/300 Write block 394 len 40960 data: 5 5 3 3 4 5 4 4 3 3 073/300 Write block 2911 len 28672 data: 2 3 4 3 3 4 5 074/300 Read block 1402 len 12288 075/300 Write block 1536 len 4096 data: 3 076/300 Read block 4705 len 40960 077/300 Write block 4665 len 40960 data: 3 3 3 2 2 2 3 3 3 3 078/300 Read block 4658 len 8192 079/300 Read block 55 len 28672 080/300 Read block 2044 len 28672 081/300 Read block 3786 len 24576 082/300 Read block 3086 len 4096 083/300 Write block 1275 len 12288 data: 4 4 4 084/300 Write block 3071 len 32768 data: 2 2 3 4 4 3 5 4 085/300 Read block 4947 len 8192 086/300 Flush 087/300 Write block 4518 len 16384 data: 5 4 4 5 088/300 Write block 2848 len 8192 data: 5 3 089/300 Read block 4487 len 12288 090/300 Read block 4549 len 24576 091/300 Read block 2096 len 12288 092/300 Read block 2502 len 40960 093/300 Write block 850 len 8192 data: 6 5 094/300 Read block 2403 len 36864 095/300 Write block 2490 len 32768 data: 4 4 4 4 3 2 2 2 096/300 Read block 4762 len 24576 097/300 Read block 3546 len 28672 098/300 Write block 1686 len 12288 data: 3 3 3 099/300 Write block 1885 len 32768 data: 5 6 4 5 4 4 3 3 100/300 Flush 101/300 Flush 102/300 Read block 2084 len 12288 103/300 Write block 2731 len 20480 data: 4 4 6 5 5 104/300 Read block 318 len 20480 105/300 Write block 1103 len 40960 data: 5 5 5 5 5 5 6 6 6 7 106/300 Write block 3477 len 32768 data: 4 4 4 3 3 3 3 3 107/300 Write block 948 len 8192 data: 5 5 108/300 Read block 3592 len 28672 109/300 Read block 197 len 4096 110/300 Write block 476 len 24576 data: 3 3 3 4 3 3 111/300 Read block 3227 len 4096 112/300 Write block 4492 len 36864 data: 4 4 4 3 3 2 3 3 3 113/300 Read block 597 len 40960 114/300 Read block 989 len 12288 115/300 Read block 4798 len 40960 116/300 Read block 186 len 32768 117/300 Read block 2718 len 40960 118/300 Flush 119/300 Write block 1412 len 20480 data: 2 2 3 3 2 120/300 Write block 4434 len 28672 data: 5 3 3 4 4 3 2 121/300 Read block 3375 len 12288 122/300 Write block 2847 len 24576 data: 5 6 4 2 3 3 123/300 Read block 1035 len 24576 124/300 Flush 125/300 Flush 126/300 Read block 3623 len 8192 127/300 Read block 658 len 28672 128/300 Write block 1070 len 24576 data: 5 4 5 5 4 4 129/300 Flush 130/300 Write block 4921 len 36864 data: 4 3 5 5 6 6 6 6 4 131/300 Write block 1039 len 12288 data: 5 7 4 132/300 Read block 2052 len 40960 133/300 Write block 2635 len 36864 data: 5 4 4 4 4 4 4 4 4 134/300 Write block 3734 len 28672 data: 2 2 4 4 4 5 5 135/300 Flush 136/300 Write block 4039 len 8192 data: 3 3 137/300 Read block 3483 len 28672 138/300 Read block 367 len 40960 139/300 Write block 2255 len 20480 data: 3 5 4 4 4 140/300 Read block 2784 len 8192 141/300 Read block 4819 len 36864 142/300 Read block 1051 len 4096 143/300 Write block 2696 len 16384 data: 5 3 3 3 144/300 Read block 4949 len 32768 145/300 Write block 4706 len 4096 data: 2 146/300 Read block 1334 len 40960 147/300 Flush 148/300 Write block 4397 len 20480 data: 2 2 4 4 4 149/300 Write block 891 len 40960 data: 4 5 4 4 4 4 5 6 5 5 150/300 Read block 547 len 28672 151/300 Write block 983 len 4096 data: 3 152/300 Write block 1399 len 28672 data: 3 3 3 3 3 3 3 153/300 Write block 2555 len 20480 data: 3 2 2 2 3 154/300 Write block 2570 len 36864 data: 3 3 3 5 5 5 5 5 5 155/300 Read block 4405 len 28672 156/300 Write block 1772 len 24576 data: 7 6 5 7 6 4 157/300 Flush 158/300 Write block 4482 len 24576 data: 2 2 2 3 2 2 159/300 Write block 1679 len 20480 data: 5 5 5 3 4 160/300 Read block 1383 len 28672 161/300 Write block 2180 len 4096 data: 2 162/300 Write block 3106 len 40960 data: 5 4 3 4 3 3 3 4 3 4 163/300 Read block 2495 len 8192 164/300 Read block 3480 len 24576 165/300 Write block 2259 len 16384 data: 5 4 4 4 166/300 Write block 2928 len 40960 data: 2 2 2 2 3 4 5 4 5 3 167/300 Write block 2925 len 4096 data: 3 168/300 Read block 458 len 24576 169/300 Read block 1723 len 32768 170/300 Read block 1119 len 40960 171/300 Flush 172/300 Flush 173/300 Read block 64 len 12288 174/300 Read block 698 len 20480 175/300 Write block 2616 len 24576 data: 4 3 4 4 4 4 176/300 Read block 726 len 40960 177/300 Read block 4266 len 8192 178/300 Read block 236 len 4096 179/300 Read block 1151 len 20480 180/300 Read block 4232 len 24576 181/300 Read block 543 len 24576 182/300 Read block 2591 len 12288 183/300 Read block 3786 len 28672 184/300 Write block 1089 len 28672 data: 2 2 2 2 2 3 3 185/300 Write block 1242 len 12288 data: 4 5 5 186/300 Read block 163 len 36864 187/300 Read block 2646 len 24576 188/300 Read block 2877 len 16384 189/300 Write block 1392 len 20480 data: 7 6 5 5 5 190/300 Read block 1076 len 36864 191/300 Write block 3548 len 16384 data: 2 3 3 3 192/300 Write block 1784 len 8192 data: 2 2 193/300 Read block 1336 len 24576 194/300 Read block 4021 len 24576 195/300 Read block 749 len 40960 196/300 Read block 1018 len 28672 197/300 Read block 4495 len 32768 198/300 Read block 2734 len 16384 199/300 Write block 1288 len 4096 data: 3 200/300 Read block 3666 len 24576 201/300 Flush 202/300 Write block 2236 len 4096 data: 5 203/300 Read block 911 len 20480 204/300 Write block 4009 len 8192 data: 5 5 205/300 Read block 551 len 4096 206/300 Write block 870 len 20480 data: 3 3 3 2 2 207/300 Write block 3193 len 32768 data: 4 3 3 2 3 2 2 2 208/300 Write block 1070 len 40960 data: 6 5 6 6 5 5 4 5 4 3 209/300 Flush 210/300 Write block 1206 len 12288 data: 3 3 3 211/300 Write block 4400 len 40960 data: 5 5 5 4 2 2 2 3 3 4 212/300 Write block 2057 len 32768 data: 4 3 3 3 3 3 2 2 213/300 Read block 4087 len 4096 214/300 Read block 3752 len 20480 215/300 Flush 216/300 Read block 1186 len 28672 217/300 Read block 4615 len 40960 218/300 Write block 4969 len 8192 data: 3 3 219/300 Read block 3654 len 16384 220/300 Flush 221/300 Write block 3229 len 36864 data: 5 5 4 4 4 3 4 5 6 222/300 Read block 930 len 12288 223/300 Write block 4935 len 8192 data: 5 4 224/300 Read block 1649 len 8192 225/300 Read block 4440 len 12288 226/300 Write block 3975 len 24576 data: 3 3 4 4 4 4 227/300 Read block 3116 len 28672 228/300 Read block 1155 len 36864 229/300 Write block 1025 len 4096 data: 4 230/300 Read block 1499 len 36864 231/300 Write block 2577 len 36864 data: 6 6 5 4 4 4 4 5 4 232/300 Flush 233/300 Read block 3834 len 12288 234/300 Read block 1811 len 20480 235/300 Flush 236/300 Write block 391 len 20480 data: 3 4 5 6 6 237/300 Read block 1291 len 28672 238/300 Read block 1664 len 12288 239/300 Read block 2980 len 32768 240/300 Write block 663 len 40960 data: 7 7 7 5 5 3 3 4 4 2 241/300 Read block 2553 len 12288 242/300 Write block 1589 len 36864 data: 3 3 5 5 4 4 4 4 4 243/300 Read block 4851 len 20480 244/300 Read block 3857 len 24576 245/300 Write block 3699 len 20480 data: 3 3 3 3 2 246/300 Flush 247/300 Write block 3705 len 20480 data: 3 3 3 2 3 248/300 Write block 1109 len 20480 data: 7 7 7 8 6 249/300 Read block 4764 len 24576 250/300 Write block 2967 len 32768 data: 5 5 5 4 2 2 2 3 251/300 Write block 4477 len 20480 data: 2 2 2 2 2 252/300 Read block 4346 len 36864 253/300 Write block 4661 len 24576 data: 2 2 2 3 4 4 254/300 Write block 2843 len 24576 data: 5 5 5 5 6 7 255/300 Read block 4742 len 24576 256/300 Write block 688 len 8192 data: 6 6 257/300 Read block 72 len 12288 258/300 Write block 2816 len 24576 data: 4 3 3 4 5 7 259/300 Write block 4580 len 20480 data: 7 6 5 4 3 260/300 Flush 261/300 Write block 323 len 36864 data: 6 4 4 3 3 3 3 3 4 262/300 Read block 96 len 40960 263/300 Read block 4759 len 32768 264/300 Read block 1672 len 40960 265/300 Read block 2984 len 32768 266/300 Write block 3254 len 24576 data: 4 4 4 4 4 6 267/300 Write block 1503 len 40960 data: 3 4 4 4 4 5 4 5 4 3 268/300 Write block 4209 len 28672 data: 4 3 3 3 4 5 7 269/300 Write block 4576 len 24576 data: 6 6 6 7 8 7 270/300 Write block 4979 len 4096 data: 5 271/300 Read block 688 len 28672 272/300 Read block 3454 len 4096 273/300 Flush 274/300 Write block 4009 len 24576 data: 6 6 5 5 5 6 275/300 Read block 2725 len 4096 276/300 Write block 3086 len 16384 data: 4 2 2 2 277/300 Write block 903 len 12288 data: 3 2 2 278/300 Read block 3874 len 12288 279/300 Write block 4908 len 28672 data: 5 4 4 4 4 4 5 280/300 Write block 36 len 40960 data: 3 2 3 4 4 4 4 3 2 2 281/300 Read block 4116 len 12288 282/300 Read block 2606 len 12288 283/300 Write block 448 len 28672 data: 2 2 2 3 3 4 3 284/300 Flush 285/300 Flush 286/300 Read block 1178 len 8192 287/300 Read block 1108 len 8192 288/300 Read block 344 len 36864 289/300 Write block 1260 len 20480 data: 5 5 3 3 3 290/300 Read block 4328 len 24576 291/300 Write block 1428 len 20480 data: 3 3 3 3 3 292/300 Read block 2658 len 8192 293/300 Write block 4337 len 36864 data: 4 3 4 4 5 5 6 6 5 294/300 Read block 4862 len 36864 295/300 Read block 4304 len 28672 296/300 Write block 829 len 28672 data: 3 2 2 2 2 4 5 297/300 Write block 2526 len 32768 data: 2 2 2 3 3 3 4 4 298/300 Read block 2026 len 16384 299/300 Read block 3638 len 28672 300/300 Read block 4776 len 12288 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:33:01.781 INFO request completed, latency_us: 349, response_code: 204, uri: /start/cid/2, method: POST, req_id: 9f13158b-3166-4936-81cd-854e4c1ccd69, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 3627 Acked 4626 Write 3 Done Done New true 3628 Acked 4627 Read 8 Done Done New true 3629 Acked 4628 Read 4 Done Done New false 3630 Acked 4629 Write 4 Done Done New false 3631 Acked 4630 Write 3 Done Done New false 3632 Acked 4631 Write 5 Done Done New false 3633 Acked 4632 Write 1 Done Done New false 3634 Acked 4633 Read 3 Done Done New false 3635 Acked 4634 Write 6 Done Done New false 3636 Acked 4635 Write 8 Done Done New false 3637 Acked 4636 Write 4 Done Done New false 3638 Acked 4637 Write 10 Done Done New false 3639 Acked 4638 Write 1 Done Done New false 3640 Acked 4639 Read 4 Done Done New false 3641 Acked 4640 Write 10 Done Done New false 3642 Acked 4641 Read 3 Done Done New false 3643 Acked 4642 Read 9 Done Done New false 3644 Acked 4643 Write 4 Done Done New false 3645 Acked 4644 Write 7 Done Done New false 3646 Acked 4645 Write 6 Done Done New false 3647 Acked 4646 Write 9 Done Done New false 3648 Acked 4647 Write 8 Done Done New false 3649 Acked 4648 Write 3 Done Done New false 3650 Acked 4649 Read 9 Done Done New false 3651 Acked 4650 Read 2 Done Done New false 3652 Acked 4651 Read 10 Done Done New false 3653 Acked 4652 Write 4 Done Done New false 3654 Acked 4653 Read 4 Done Done New false 3655 Acked 4654 Flush 0 Done Done New false 3656 Acked 4655 Read 3 Done Done New false 3657 Acked 4656 Write 9 Done Done New false 3658 Acked 4657 Write 6 Done Done New false 3659 Acked 4658 Read 8 Done Done New false 3660 Acked 4659 Write 9 Done Done New false 3661 Acked 4660 Read 8 Done Done New false 3662 Acked 4661 Write 10 Done Done New false 3663 Acked 4662 Write 8 Done Done New false 3664 Acked 4663 Read 10 Done Done New false 3665 Acked 4664 Write 10 Done Done New false 3666 Acked 4665 Read 1 Done Done New false 3667 Acked 4666 Read 9 Done Done New false 3668 Acked 4667 Write 2 Done Done New false 3669 Acked 4668 Write 8 Done Done New false 3670 Acked 4669 Read 2 Done Done New false 3671 Acked 4670 Read 8 Done Done New false 3672 Acked 4671 Write 4 Done Done New false 3673 Acked 4672 Read 5 Done Done New false 3674 Acked 4673 Read 5 Done Done New false 3675 Acked 4674 Flush 0 Done Done New false 3676 Acked 4675 Read 10 Done Done New false 3677 Acked 4676 Write 1 Done Done New false 3678 Acked 4677 Read 10 Done Done New false 3679 Acked 4678 Write 9 Done Done New false 3680 Acked 4679 Write 4 Done Done New false 3681 Acked 4680 Read 10 Done Done New false 3682 Acked 4681 Read 6 Done Done New false 3683 Acked 4682 Read 4 Done Done New false 3684 Acked 4683 Read 7 Done Done New false 3685 Acked 4684 Read 6 Done Done New false 3686 Acked 4685 Read 7 Done Done New false 3687 Acked 4686 Write 4 Done Done New false 3688 Acked 4687 Write 6 Done Done New false 3689 Acked 4688 Write 9 Done Done New false 3690 Acked 4689 Read 10 Done Done New false 3691 Acked 4690 Write 7 Done Done New false 3692 Acked 4691 Write 1 Done Done New false 3693 Acked 4692 Read 8 Done Done New false 3694 Acked 4693 Write 2 Done Done New false 3695 Acked 4694 Read 1 Done Done New false 3696 Acked 4695 Flush 0 Done Done New false 3697 Acked 4696 Read 6 Done Done New false 3698 Acked 4697 Write 10 Done Done New false 3699 Acked 4698 Write 7 Done Done New false 3700 Acked 4699 Read 3 Done Done New false 3701 Acked 4700 Write 1 Done Done New false 3702 Acked 4701 Read 10 Done Done New false 3703 Acked 4702 Write 10 Done Done New false 3704 Acked 4703 Read 2 Done Done New false 3705 Acked 4704 Read 7 Done Done New false 3706 Acked 4705 Read 7 Done Done New false 3707 Acked 4706 Read 6 Done Done New false 3708 Acked 4707 Read 1 Done Done New false 3709 Acked 4708 Write 3 Done Done New false 3710 Acked 4709 Write 8 Done Done New false 3711 Acked 4710 Read 2 Done Done New false 3712 Acked 4711 Flush 0 Done Done New false 3713 Acked 4712 Write 4 Done Done New false 3714 Acked 4713 Write 2 Done Done New false 3715 Acked 4714 Read 3 Done Done New false 3716 Acked 4715 Read 6 Done Done New false 3717 Acked 4716 Read 3 Done Done New false 3718 Acked 4717 Read 10 Done Done New false 3719 Acked 4718 Write 2 Done Done New false 3720 Acked 4719 Read 9 Done Done New false 3721 Acked 4720 Write 8 Done Done New false 3722 Acked 4721 Read 6 Done Done New false 3723 Acked 4722 Read 7 Done Done New false 3724 Acked 4723 Write 3 Done Done New false 3725 Acked 4724 Write 8 Done Done New false 3726 Acked 4725 Flush 0 Done Done New false 3727 Acked 4726 Flush 0 Done Done New false 3728 Acked 4727 Read 3 Done Done New false 3729 Acked 4728 Write 5 Done Done New false 3730 Acked 4729 Read 5 Done Done New false 3731 Acked 4730 Write 10 Done Done New false 3732 Acked 4731 Write 8 Done Done New false 3733 Acked 4732 Write 2 Done Done New false 3734 Acked 4733 Read 7 Done Done New false 3735 Acked 4734 Read 1 Done Done New false 3736 Acked 4735 Write 6 Done Done New false 3737 Acked 4736 Read 1 Done Done New false 3738 Acked 4737 Write 9 Done Done New false 3739 Acked 4738 Read 10 Done Done New false 3740 Acked 4739 Read 3 Done Done New false 3741 Acked 4740 Read 10 Done Done New false 3742 Acked 4741 Read 8 Done Done New false 3743 Acked 4742 Read 10 Done Done New false 3744 Acked 4743 Flush 0 Done Done New false 3745 Acked 4744 Write 5 Done Done New false 3746 Acked 4745 Write 7 Done Done New false 3747 Acked 4746 Read 3 Done Done New false 3748 Acked 4747 Write 6 Done Done New false 3749 Acked 4748 Read 6 Done Done New false 3750 Acked 4749 Flush 0 Done Done New false 3751 Acked 4750 Flush 0 Done Done New false 3752 Acked 4751 Read 2 Done Done New false 3753 Acked 4752 Read 7 Done Done New false 3754 Acked 4753 Write 6 Done Done New false 3755 Acked 4754 Flush 0 Done Done New false 3756 Acked 4755 Write 9 Done Done New false 3757 Acked 4756 Write 3 Done Done New false 3758 Acked 4757 Read 10 Done Done New false 3759 Acked 4758 Write 9 Done Done New false 3760 Acked 4759 Write 7 Done Done New false 3761 Acked 4760 Flush 0 Done Done New false 3762 Acked 4761 Write 2 Done Done New false 3763 Acked 4762 Read 7 Done Done New false 3764 Acked 4763 Read 10 Done Done New false 3765 Acked 4764 Write 5 Done Done New false 3766 Acked 4765 Read 2 Done Done New false 3767 Acked 4766 Read 9 Done Done New false 3768 Acked 4767 Read 1 Done Done New false 3769 Acked 4768 Write 4 Done Done New false 3770 Acked 4769 Read 8 Done Done New false 3771 Acked 4770 Write 1 Done Done New false 3772 Acked 4771 Read 10 Done Done New false 3773 Acked 4772 Flush 0 Done Done New false 3774 Acked 4773 Write 5 Done Done New false 3775 Acked 4774 Write 10 Done Done New false 3776 Acked 4775 Read 7 Done Done New false 3777 Acked 4776 Write 1 Done Done New false 3778 Acked 4777 Write 7 Done Done New false 3779 Acked 4778 Write 5 Done Done New false 3780 Acked 4779 Write 9 Done Done New false 3781 Acked 4780 Read 7 Done Done New false 3782 Acked 4781 Write 6 Done Done New false 3783 Acked 4782 Flush 0 Done Done New false 3784 Acked 4783 Write 6 Done Done New false 3785 Acked 4784 Write 5 Done Done New false 3786 Acked 4785 Read 7 Done Done New false 3787 Acked 4786 Write 1 Done Done New false 3788 Acked 4787 Write 10 Done Done New false 3789 Acked 4788 Read 2 Done Done New false 3790 Acked 4789 Read 6 Done Done New false 3791 Acked 4790 Write 4 Done Done New false 3792 Acked 4791 Write 10 Done Done New false 3793 Acked 4792 Write 1 Done Done New false 3794 Acked 4793 Read 6 Done Done New false 3795 Acked 4794 Read 8 Done Done New false 3796 Acked 4795 Read 10 Done Done New false 3797 Acked 4796 Flush 0 Done Done New false 3798 Acked 4797 Flush 0 Done Done New false 3799 Acked 4798 Read 3 Done Done New false 3800 Acked 4799 Read 5 Done Done New false 3801 Acked 4800 Write 6 Done Done New false 3802 Acked 4801 Read 10 Done Done New false 3803 Acked 4802 Read 2 Done Done New false 3804 Acked 4803 Read 1 Done Done New false 3805 Acked 4804 Read 5 Done Done New false 3806 Acked 4805 Read 6 Done Done New false 3807 Acked 4806 Read 6 Done Done New false 3808 Acked 4807 Read 3 Done Done New false 3809 Acked 4808 Read 7 Done Done New false 3810 Acked 4809 Write 7 Done Done New false 3811 Acked 4810 Write 3 Done Done New false 3812 Acked 4811 Read 9 Done Done New false 3813 Acked 4812 Read 6 Done Done New false 3814 Acked 4813 Read 4 Done Done New false 3815 Acked 4814 Write 5 Done Done New false 3816 Acked 4815 Read 9 Done Done New false 3817 Acked 4816 Write 4 Done Done New false 3818 Acked 4817 Write 2 Done Done New false 3819 Acked 4818 Read 6 Done Done New false 3820 Acked 4819 Read 6 Done Done New false 3821 Acked 4820 Read 10 Done Done New false 3822 Acked 4821 Read 7 Done Done New false 3823 Acked 4822 Read 8 Done Done New false 3824 Acked 4823 Read 4 Done Done New false 3825 Acked 4824 Write 1 Done Done New false 3826 Acked 4825 Read 6 Done Done New false 3827 Acked 4826 Flush 0 Done Done New false 3828 Acked 4827 Write 1 Done Done New false 3829 Acked 4828 Read 5 Done Done New false 3830 Acked 4829 Write 2 Done Done New false 3831 Acked 4830 Read 1 Done Done New false 3832 Acked 4831 Flush 0 Done Done New false 3833 Acked 4832 Write 5 Done Done New false 3834 Acked 4833 Write 8 Done Done New false 3835 Acked 4834 Write 10 Done Done New false 3836 Acked 4835 Flush 0 Done Done New false 3837 Acked 4836 Write 3 Done Done New false 3838 Acked 4837 Write 10 Done Done New false 3839 Acked 4838 Write 8 Done Done New false 3840 Acked 4839 Read 1 Done Done New false 3841 Acked 4840 Read 5 Done Done New false 3842 Acked 4841 Flush 0 Done Done New false 3843 Acked 4842 Read 7 Done Done New false 3844 Acked 4843 Read 10 Done Done New false 3845 Acked 4844 Write 2 Done Done New false 3846 Acked 4845 Read 4 Done Done New false 3847 Acked 4846 Flush 0 Done Done New false 3848 Acked 4847 Write 9 Done Done New false 3849 Acked 4848 Read 3 Done Done New false 3850 Acked 4849 Write 2 Done Done New false 3851 Acked 4850 Read 2 Done Done New false 3852 Acked 4851 Read 3 Done Done New false 3853 Acked 4852 Write 6 Done Done New false 3854 Acked 4853 Read 7 Done Done New false 3855 Acked 4854 Read 9 Done Done New false 3856 Acked 4855 Write 1 Done Done New false 3857 Acked 4856 Read 9 Done Done New false 3858 Acked 4857 Write 9 Done Done New false 3859 Acked 4858 Flush 0 Done Done New false 3860 Acked 4859 Read 3 Done Done New false 3861 Acked 4860 Read 5 Done Done New false 3862 Acked 4861 Flush 0 Done Done New false 3863 Acked 4862 Write 5 Done Done New false 3864 Acked 4863 Read 7 Done Done New false 3865 Acked 4864 Read 3 Done Done New false 3866 Acked 4865 Read 8 Done Done New false 3867 Acked 4866 Write 10 Done Done New false 3868 Acked 4867 Read 3 Done Done New false 3869 Acked 4868 Write 9 Done Done New false 3870 Acked 4869 Read 5 Done Done New false 3871 Acked 4870 Read 6 Done Done New false 3872 Acked 4871 Write 5 Done Done New false 3873 Acked 4872 Flush 0 Done Done New false 3874 Acked 4873 Write 5 Done Done New false 3875 Acked 4874 Write 5 Done Done New false 3876 Acked 4875 Read 6 Done Done New false 3877 Acked 4876 Write 8 Done Done New false 3878 Acked 4877 Write 5 Done Done New false 3879 Acked 4878 Read 9 Done Done New false 3880 Acked 4879 Write 6 Done Done New false 3881 Acked 4880 Write 6 Done Done New false 3882 Acked 4881 Read 6 Done Done New false 3883 Acked 4882 Write 2 Done Done New false 3884 Acked 4883 Read 3 Done Done New false 3885 Acked 4884 Write 6 Done Done New false 3886 Acked 4885 Write 5 Done Done New false 3887 Acked 4886 Flush 0 Done Done New false 3888 Acked 4887 Write 9 Done Done New false 3889 Acked 4888 Read 10 Done Done New false 3890 Acked 4889 Read 8 Done Done New false 3891 Acked 4890 Read 10 Done Done New false 3892 Acked 4891 Read 8 Done Done New false 3893 Acked 4892 Write 6 Done Done New false 3894 Acked 4893 Write 10 Done Done New false 3895 Acked 4894 Write 7 Done Done New false 3896 Acked 4895 Write 6 Done Done New false 3897 Acked 4896 Write 1 Done Done New false 3898 Acked 4897 Read 7 Done Done New false 3899 Acked 4898 Read 1 Done Done New false 3900 Acked 4899 Flush 0 Done Done New false 3901 Acked 4900 Write 6 Done Done New false 3902 Acked 4901 Read 1 Done Done New false 3903 Acked 4902 Write 4 Done Done New false 3904 Acked 4903 Write 3 Done Done New false 3905 Acked 4904 Read 3 Done Done New false 3906 Acked 4905 Write 7 Done Done New false 3907 Acked 4906 Write 10 Done Done New false 3908 Acked 4907 Read 3 Done Done New false 3909 Acked 4908 Read 3 Done Done New false 3910 Acked 4909 Write 7 Done Done New false 3911 Acked 4910 Flush 0 Done Done New false 3912 Acked 4911 Flush 0 Done Done New false 3913 Acked 4912 Read 2 Done Done New false 3914 Acked 4913 Read 2 Done Done New false 3915 Acked 4914 Read 9 Done Done New false 3916 Acked 4915 Write 5 Done Done New false 3917 Acked 4916 Read 6 Done Done New false 3918 Acked 4917 Write 5 Done Done New false 3919 Acked 4918 Read 2 Done Done New false 3920 Acked 4919 Write 9 Done Done New false 3921 Acked 4920 Read 9 Done Done New false 3922 Acked 4921 Read 7 Done Done New false 3923 Acked 4922 Write 7 Done Done New false 3924 Acked 4923 Write 8 Done Done New false 3925 Acked 4924 Read 4 Done Done New false 3926 Acked 4925 Read 7 Done Done New false 3927 Acked 4926 Read 3 Done Done New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 301 301 Sent 0 0 0 0 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4911 4911 4625 Downstairs last five completed: 4625 4624 4623 4622 4621 Upstairs last five completed: 3927 3926 3925 3924 3923 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:02.148501343Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:02.148572125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:02.148600684Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:02.148634259Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:02.148661644Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:02.148682028Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:02.148704622Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:02.148925052Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:03.150059823Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:03.15012607Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:03.150153785Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:03.150180256Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:03.150200643Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:03.150233645Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:03.150472257Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:04.151379998Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:04.151444924Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:04.151482429Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:04.151509727Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:04.15153011Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:04.151552822Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:04.151789037Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:05.152725316Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:05.152851797Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:05.152892751Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:05.152921485Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:05.152941821Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:05.152997368Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:05.153250096Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 3627 Acked 4626 Write 3 Done Done New true 3628 Acked 4627 Read 8 Done Done New true 3629 Acked 4628 Read 4 Done Done New true 3630 Acked 4629 Write 4 Done Done New true 3631 Acked 4630 Write 3 Done Done New true 3632 Acked 4631 Write 5 Done Done New true 3633 Acked 4632 Write 1 Done Done New true 3634 Acked 4633 Read 3 Done Done New true 3635 Acked 4634 Write 6 Done Done New true 3636 Acked 4635 Write 8 Done Done New true 3637 Acked 4636 Write 4 Done Done New true 3638 Acked 4637 Write 10 Done Done New true 3639 Acked 4638 Write 1 Done Done New true 3640 Acked 4639 Read 4 Done Done New true 3641 Acked 4640 Write 10 Done Done New true 3642 Acked 4641 Read 3 Done Done New true 3643 Acked 4642 Read 9 Done Done New true 3644 Acked 4643 Write 4 Done Done New true 3645 Acked 4644 Write 7 Done Done New true 3646 Acked 4645 Write 6 Done Done New true 3647 Acked 4646 Write 9 Done Done New true 3648 Acked 4647 Write 8 Done Done New true 3649 Acked 4648 Write 3 Done Done New true 3650 Acked 4649 Read 9 Done Done New true 3651 Acked 4650 Read 2 Done Done New true 3652 Acked 4651 Read 10 Done Done New true 3653 Acked 4652 Write 4 Done Done New true 3654 Acked 4653 Read 4 Done Done New true 3655 Acked 4654 Flush 0 Done Done New true 3656 Acked 4655 Read 3 Done Done New true 3657 Acked 4656 Write 9 Done Done New true 3658 Acked 4657 Write 6 Done Done New true 3659 Acked 4658 Read 8 Done Done New true 3660 Acked 4659 Write 9 Done Done New true 3661 Acked 4660 Read 8 Done Done New true 3662 Acked 4661 Write 10 Done Done New true 3663 Acked 4662 Write 8 Done Done New true 3664 Acked 4663 Read 10 Done Done New true 3665 Acked 4664 Write 10 Done Done New true 3666 Acked 4665 Read 1 Done Done New true 3667 Acked 4666 Read 9 Done Done New true 3668 Acked 4667 Write 2 Done Done New true 3669 Acked 4668 Write 8 Done Done New true 3670 Acked 4669 Read 2 Done Done New true 3671 Acked 4670 Read 8 Done Done New true 3672 Acked 4671 Write 4 Done Done New true 3673 Acked 4672 Read 5 Done Done New true 3674 Acked 4673 Read 5 Done Done New true 3675 Acked 4674 Flush 0 Done Done New true 3676 Acked 4675 Read 10 Done Done New true 3677 Acked 4676 Write 1 Done Done New true 3678 Acked 4677 Read 10 Done Done New true 3679 Acked 4678 Write 9 Done Done New true 3680 Acked 4679 Write 4 Done Done New true 3681 Acked 4680 Read 10 Done Done New true 3682 Acked 4681 Read 6 Done Done New true 3683 Acked 4682 Read 4 Done Done New true 3684 Acked 4683 Read 7 Done Done New true 3685 Acked 4684 Read 6 Done Done New true 3686 Acked 4685 Read 7 Done Done New true 3687 Acked 4686 Write 4 Done Done New true 3688 Acked 4687 Write 6 Done Done New true 3689 Acked 4688 Write 9 Done Done New true 3690 Acked 4689 Read 10 Done Done New true 3691 Acked 4690 Write 7 Done Done New true 3692 Acked 4691 Write 1 Done Done New true 3693 Acked 4692 Read 8 Done Done New true 3694 Acked 4693 Write 2 Done Done New true 3695 Acked 4694 Read 1 Done Done New true 3696 Acked 4695 Flush 0 Done Done New true 3697 Acked 4696 Read 6 Done Done New true 3698 Acked 4697 Write 10 Done Done New true 3699 Acked 4698 Write 7 Done Done New true 3700 Acked 4699 Read 3 Done Done New true 3701 Acked 4700 Write 1 Done Done New true 3702 Acked 4701 Read 10 Done Done New true 3703 Acked 4702 Write 10 Done Done New true 3704 Acked 4703 Read 2 Done Done New true 3705 Acked 4704 Read 7 Done Done New true 3706 Acked 4705 Read 7 Done Done New true 3707 Acked 4706 Read 6 Done Done New true 3708 Acked 4707 Read 1 Done Done New true 3709 Acked 4708 Write 3 Done Done New true 3710 Acked 4709 Write 8 Done Done New true 3711 Acked 4710 Read 2 Done Done New true 3712 Acked 4711 Flush 0 Done Done New true 3713 Acked 4712 Write 4 Done Done New true 3714 Acked 4713 Write 2 Done Done New true 3715 Acked 4714 Read 3 Done Done New true 3716 Acked 4715 Read 6 Done Done New true 3717 Acked 4716 Read 3 Done Done New true 3718 Acked 4717 Read 10 Done Done New true 3719 Acked 4718 Write 2 Done Done New true 3720 Acked 4719 Read 9 Done Done New true 3721 Acked 4720 Write 8 Done Done New true 3722 Acked 4721 Read 6 Done Done New true 3723 Acked 4722 Read 7 Done Done New true 3724 Acked 4723 Write 3 Done Done New true 3725 Acked 4724 Write 8 Done Done New true 3726 Acked 4725 Flush 0 Done Done New true 3727 Acked 4726 Flush 0 Done Done New true 3728 Acked 4727 Read 3 Done Done New true 3729 Acked 4728 Write 5 Done Done New true 3730 Acked 4729 Read 5 Done Done New true 3731 Acked 4730 Write 10 Done Done New true 3732 Acked 4731 Write 8 Done Done New true 3733 Acked 4732 Write 2 Done Done New true 3734 Acked 4733 Read 7 Done Done New true 3735 Acked 4734 Read 1 Done Done New true 3736 Acked 4735 Write 6 Done Done New true 3737 Acked 4736 Read 1 Done Done New true 3738 Acked 4737 Write 9 Done Done New true 3739 Acked 4738 Read 10 Done Done New true 3740 Acked 4739 Read 3 Done Done New true 3741 Acked 4740 Read 10 Done Done New true 3742 Acked 4741 Read 8 Done Done New true 3743 Acked 4742 Read 10 Done Done New true 3744 Acked 4743 Flush 0 Done Done New true 3745 Acked 4744 Write 5 Done Done New true 3746 Acked 4745 Write 7 Done Done New true 3747 Acked 4746 Read 3 Done Done New true 3748 Acked 4747 Write 6 Done Done New true 3749 Acked 4748 Read 6 Done Done New true 3750 Acked 4749 Flush 0 Done Done New true 3751 Acked 4750 Flush 0 Done Done New true 3752 Acked 4751 Read 2 Done Done New true 3753 Acked 4752 Read 7 Done Done New true 3754 Acked 4753 Write 6 Done Done New true 3755 Acked 4754 Flush 0 Done Done New true 3756 Acked 4755 Write 9 Done Done New true 3757 Acked 4756 Write 3 Done Done New true 3758 Acked 4757 Read 10 Done Done New true 3759 Acked 4758 Write 9 Done Done New true 3760 Acked 4759 Write 7 Done Done New true 3761 Acked 4760 Flush 0 Done Done New true 3762 Acked 4761 Write 2 Done Done New true 3763 Acked 4762 Read 7 Done Done New true 3764 Acked 4763 Read 10 Done Done New true 3765 Acked 4764 Write 5 Done Done New true 3766 Acked 4765 Read 2 Done Done New true 3767 Acked 4766 Read 9 Done Done New true 3768 Acked 4767 Read 1 Done Done New true 3769 Acked 4768 Write 4 Done Done New true 3770 Acked 4769 Read 8 Done Done New true 3771 Acked 4770 Write 1 Done Done New true 3772 Acked 4771 Read 10 Done Done New true 3773 Acked 4772 Flush 0 Done Done New true 3774 Acked 4773 Write 5 Done Done New true 3775 Acked 4774 Write 10 Done Done New true 3776 Acked 4775 Read 7 Done Done New true 3777 Acked 4776 Write 1 Done Done New true 3778 Acked 4777 Write 7 Done Done New true 3779 Acked 4778 Write 5 Done Done New true 3780 Acked 4779 Write 9 Done Done New true 3781 Acked 4780 Read 7 Done Done New true 3782 Acked 4781 Write 6 Done Done New true 3783 Acked 4782 Flush 0 Done Done New true 3784 Acked 4783 Write 6 Done Done New true 3785 Acked 4784 Write 5 Done Done New true 3786 Acked 4785 Read 7 Done Done New true 3787 Acked 4786 Write 1 Done Done New true 3788 Acked 4787 Write 10 Done Done New true 3789 Acked 4788 Read 2 Done Done New true 3790 Acked 4789 Read 6 Done Done New true 3791 Acked 4790 Write 4 Done Done New true 3792 Acked 4791 Write 10 Done Done New true 3793 Acked 4792 Write 1 Done Done New true 3794 Acked 4793 Read 6 Done Done New true 3795 Acked 4794 Read 8 Done Done New true 3796 Acked 4795 Read 10 Done Done New true 3797 Acked 4796 Flush 0 Done Done New true 3798 Acked 4797 Flush 0 Done Done New true 3799 Acked 4798 Read 3 Done Done New true 3800 Acked 4799 Read 5 Done Done New true 3801 Acked 4800 Write 6 Done Done New true 3802 Acked 4801 Read 10 Done Done New true 3803 Acked 4802 Read 2 Done Done New true 3804 Acked 4803 Read 1 Done Done New true 3805 Acked 4804 Read 5 Done Done New true 3806 Acked 4805 Read 6 Done Done New true 3807 Acked 4806 Read 6 Done Done New true 3808 Acked 4807 Read 3 Done Done New true 3809 Acked 4808 Read 7 Done Done New true 3810 Acked 4809 Write 7 Done Done New true 3811 Acked 4810 Write 3 Done Done New true 3812 Acked 4811 Read 9 Done Done New true 3813 Acked 4812 Read 6 Done Done New true 3814 Acked 4813 Read 4 Done Done New true 3815 Acked 4814 Write 5 Done Done New true 3816 Acked 4815 Read 9 Done Done New true 3817 Acked 4816 Write 4 Done Done New true 3818 Acked 4817 Write 2 Done Done New true 3819 Acked 4818 Read 6 Done Done New true 3820 Acked 4819 Read 6 Done Done New true 3821 Acked 4820 Read 10 Done Done New true 3822 Acked 4821 Read 7 Done Done New true 3823 Acked 4822 Read 8 Done Done New true 3824 Acked 4823 Read 4 Done Done New true 3825 Acked 4824 Write 1 Done Done New true 3826 Acked 4825 Read 6 Done Done New true 3827 Acked 4826 Flush 0 Done Done New true 3828 Acked 4827 Write 1 Done Done New true 3829 Acked 4828 Read 5 Done Done New true 3830 Acked 4829 Write 2 Done Done New true 3831 Acked 4830 Read 1 Done Done New true 3832 Acked 4831 Flush 0 Done Done New true 3833 Acked 4832 Write 5 Done Done New true 3834 Acked 4833 Write 8 Done Done New true 3835 Acked 4834 Write 10 Done Done New true 3836 Acked 4835 Flush 0 Done Done New true 3837 Acked 4836 Write 3 Done Done New true 3838 Acked 4837 Write 10 Done Done New true 3839 Acked 4838 Write 8 Done Done New true 3840 Acked 4839 Read 1 Done Done New true 3841 Acked 4840 Read 5 Done Done New true 3842 Acked 4841 Flush 0 Done Done New true 3843 Acked 4842 Read 7 Done Done New true 3844 Acked 4843 Read 10 Done Done New true 3845 Acked 4844 Write 2 Done Done New true 3846 Acked 4845 Read 4 Done Done New true 3847 Acked 4846 Flush 0 Done Done New true 3848 Acked 4847 Write 9 Done Done New true 3849 Acked 4848 Read 3 Done Done New true 3850 Acked 4849 Write 2 Done Done New true 3851 Acked 4850 Read 2 Done Done New true 3852 Acked 4851 Read 3 Done Done New true 3853 Acked 4852 Write 6 Done Done New true 3854 Acked 4853 Read 7 Done Done New true 3855 Acked 4854 Read 9 Done Done New true 3856 Acked 4855 Write 1 Done Done New true 3857 Acked 4856 Read 9 Done Done New true 3858 Acked 4857 Write 9 Done Done New true 3859 Acked 4858 Flush 0 Done Done New true 3860 Acked 4859 Read 3 Done Done New true 3861 Acked 4860 Read 5 Done Done New true 3862 Acked 4861 Flush 0 Done Done New true 3863 Acked 4862 Write 5 Done Done New true 3864 Acked 4863 Read 7 Done Done New true 3865 Acked 4864 Read 3 Done Done New true 3866 Acked 4865 Read 8 Done Done New true 3867 Acked 4866 Write 10 Done Done New true 3868 Acked 4867 Read 3 Done Done New true 3869 Acked 4868 Write 9 Done Done New true 3870 Acked 4869 Read 5 Done Done New true 3871 Acked 4870 Read 6 Done Done New true 3872 Acked 4871 Write 5 Done Done New true 3873 Acked 4872 Flush 0 Done Done New true 3874 Acked 4873 Write 5 Done Done New true 3875 Acked 4874 Write 5 Done Done New true 3876 Acked 4875 Read 6 Done Done New true 3877 Acked 4876 Write 8 Done Done New true 3878 Acked 4877 Write 5 Done Done New true 3879 Acked 4878 Read 9 Done Done New true 3880 Acked 4879 Write 6 Done Done New true 3881 Acked 4880 Write 6 Done Done New true 3882 Acked 4881 Read 6 Done Done New true 3883 Acked 4882 Write 2 Done Done New true 3884 Acked 4883 Read 3 Done Done New true 3885 Acked 4884 Write 6 Done Done New true 3886 Acked 4885 Write 5 Done Done New true 3887 Acked 4886 Flush 0 Done Done New true 3888 Acked 4887 Write 9 Done Done New true 3889 Acked 4888 Read 10 Done Done New true 3890 Acked 4889 Read 8 Done Done New true 3891 Acked 4890 Read 10 Done Done New true 3892 Acked 4891 Read 8 Done Done New true 3893 Acked 4892 Write 6 Done Done New true 3894 Acked 4893 Write 10 Done Done New true 3895 Acked 4894 Write 7 Done Done New true 3896 Acked 4895 Write 6 Done Done New true 3897 Acked 4896 Write 1 Done Done New true 3898 Acked 4897 Read 7 Done Done New true 3899 Acked 4898 Read 1 Done Done New true 3900 Acked 4899 Flush 0 Done Done New true 3901 Acked 4900 Write 6 Done Done New true 3902 Acked 4901 Read 1 Done Done New true 3903 Acked 4902 Write 4 Done Done New true 3904 Acked 4903 Write 3 Done Done New true 3905 Acked 4904 Read 3 Done Done New true 3906 Acked 4905 Write 7 Done Done New true 3907 Acked 4906 Write 10 Done Done New true 3908 Acked 4907 Read 3 Done Done New true 3909 Acked 4908 Read 3 Done Done New true 3910 Acked 4909 Write 7 Done Done New true 3911 Acked 4910 Flush 0 Done Done New true 3912 Acked 4911 Flush 0 Done Done New true 3913 Acked 4912 Read 2 Done Done New true 3914 Acked 4913 Read 2 Done Done New true 3915 Acked 4914 Read 9 Done Done New true 3916 Acked 4915 Write 5 Done Done New true 3917 Acked 4916 Read 6 Done Done New true 3918 Acked 4917 Write 5 Done Done New true 3919 Acked 4918 Read 2 Done Done New true 3920 Acked 4919 Write 9 Done Done New true 3921 Acked 4920 Read 9 Done Done New true 3922 Acked 4921 Read 7 Done Done New true 3923 Acked 4922 Write 7 Done Done New true 3924 Acked 4923 Write 8 Done Done New true 3925 Acked 4924 Read 4 Done Done New true 3926 Acked 4925 Read 7 Done Done New true 3927 Acked 4926 Read 3 Done Done New true 3928 Acked 4927 Flush 0 Done Done New true STATES DS:0 DS:1 DS:2 TOTAL New 0 0 302 302 Sent 0 0 0 0 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 4927 4927 4625 Downstairs last five completed: 4625 4624 4623 4622 4621 Upstairs last five completed: 3928 3927 3926 3925 3924 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:06.154798541Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:06.15486418Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:06.154892661Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:06.154919389Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:06.154950359Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:06.154974208Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:06.155212634Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1198) [8830][2] reports Running {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:07.156972291Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:07.157040627Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:07.157069245Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:07.157093509Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:07.157119722Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:07.157140258Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:07.157181755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:07.157437333Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.158293845Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.15846111Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.160363521Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.160404521Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.160428799Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 4625","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.160707518Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.160741412Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.160764952Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.16078723Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:08.160813634Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 4927 4926 4925 4924 4923 Upstairs last five completed: 3928 3927 3926 3925 3924 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(1) stop 1 Jan 02 22:33:09.789 INFO request completed, latency_us: 450, response_code: 204, uri: /stop/cid/1, method:[8820] Got stop action so:false kr:false POST, req_id: c40ffaf6-a2d7-41b0-850e-b9a36d37dbb0, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:33:09.790 INFO request completed, latency_us: 232, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1ec51734-d594-49ff-9f49-8da689142977, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:09.790447899Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:09.790537031Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:09.790648676Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:09.790688933Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:09.790720626Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:09.790744015Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:10.791951019Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:10.792065316Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:10.792126678Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:10.792181152Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:10.792201534Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:10.792223987Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:10.792265262Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:11.79364039Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:11.793709285Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:11.793737402Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:11.793761506Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:11.793796941Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:11.793818079Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:11.793840393Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:11.793880957Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:12.794983161Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:12.795054914Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:12.795082873Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:12.795109333Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:12.795129755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:12.795234734Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:12.795339728Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:33:13.792 INFO request completed, latency_us: 274, response_code: 200, uri: /state/cid/1, method: GET, req_id: d2bb51cd-2484-4eec-8df7-8207c92e8235, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 3201 len 4096 data: 2 002/300 Write block 817 len 16384 data: 4 4 4 3 003/300 Write block 4686 len 20480 data: 3 2 2 2 2 004/300 Flush {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:13.796607429Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:13.796963152Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:13.797003329Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:13.797031509Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:13.797051939Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 4 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:13.797074561Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:13.797096643Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 005/300 Write block 1224 len 40960 data: 4 4 4 4 5 5 4 4 3 2 006/300 Read block 4120 len 16384 007/300 Read block 3754 len 24576 008/300 Read block 4869 len 36864 009/300 Flush 010/300 Read block 1613 len 28672 011/300 Flush 012/300 Write block 1346 len 4096 data: 5 013/300 Write block 2579 len 16384 data: 6 5 5 5 014/300 Write block 2894 len 24576 data: 3 3 4 4 5 4 015/300 Write block 1645 len 8192 data: 4 3 016/300 Read block 4077 len 28672 017/300 Write block 3125 len 24576 data: 3 3 3 2 2 2 018/300 Read block 4798 len 28672 019/300 Read block 2458 len 8192 020/300 Read block 1286 len 28672 021/300 Write block 472 len 32768 data: 3 3 4 4 4 4 4 5 022/300 Read block 2349 len 36864 023/300 Write block 3086 len 8192 data: 5 3 024/300 Flush 025/300 Write block 2911 len 12288 data: 3 4 5 026/300 Write block 2125 len 40960 data: 2 2 2 2 3 3 2 2 2 2 027/300 Read block 4283 len 4096 028/300 Read block 3469 len 12288 029/300 Flush 030/300 Read block 2789 len 28672 031/300 Write block 4553 len 24576 data: 3 3 2 3 3 3 032/300 Write block 4118 len 40960 data: 3 3 3 3 3 3 4 4 4 6 033/300 Flush 034/300 Write block 2275 len 36864 data: 4 5 6 4 3 3 3 4 4 035/300 Read block 1069 len 32768 036/300 Read block 2392 len 20480 037/300 Write block 4498 len 28672 data: 4 4 4 3 3 4 4 038/300 Read block 2942 len 16384 039/300 Read block 1195 len 40960 040/300 Read block 0 len 12288 041/300 Read block 3363 len 20480 042/300 Write block 1265 len 36864 data: 3 2 2 2 2 3 3 3 3 043/300 Read block 3262 len 24576 044/300 Read block 3464 len 24576 045/300 Read block 1207 len 8192 046/300 Read block 2214 len 40960 047/300 Read block 2886 len 4096 048/300 Read block 2673 len 16384 049/300 Flush 050/300 Write block 1900 len 20480 data: 2 3 3 3 3 051/300 Read block 4632 len 36864 052/300 Read block 1501 len 36864 053/300 Write block 3027 len 12288 data: 3 3 3 054/300 Write block 4934 len 24576 data: 5 6 5 3 4 2 055/300 Read block 2784 len 24576 056/300 Read block 3872 len 40960 057/300 Write block 975 len 8192 data: 5 4 058/300 Flush 059/300 Write block 4612 len 32768 data: 5 5 5 4 5 4 2 2 060/300 Read block 3949 len 4096 061/300 Write block 1281 len 32768 data: 3 3 4 4 3 3 3 4 062/300 Read block 516 len 16384 063/300 Write block 4474 len 24576 data: 3 3 2 3 3 3 064/300 Flush 065/300 Write block 2647 len 20480 data: 3 3 3 3 5 066/300 Write block 121 len 8192 data: 3 3 067/300 Read block 1766 len 12288 068/300 Read block 2556 len 24576 069/300 Write block 4078 len 20480 data: 3 3 4 4 5 070/300 Write block 3437 len 16384 data: 2 2 2 2 071/300 Read block 1995 len 36864 072/300 Flush 073/300 Read block 4728 len 12288 074/300 Read block 1078 len 4096 075/300 Write block 2635 len 24576 data: 6 5 5 5 5 5 076/300 Read block 797 len 12288 077/300 Write block 1631 len 36864 data: 4 3 3 3 3 3 2 2 4 078/300 Write block 2001 len 4096 data: 6 079/300 Write block 4583 len 40960 data: 5 4 3 3 3 2 2 2 2 2 080/300 Read block 4693 len 24576 081/300 Read block 1207 len 4096 082/300 Read block 2366 len 4096 083/300 Read block 921 len 12288 084/300 Read block 4153 len 12288 085/300 Write block 1555 len 4096 data: 2 086/300 Write block 2194 len 20480 data: 3 3 6 5 4 087/300 Read block 4935 len 24576 088/300 Read block 1866 len 12288 089/300 Read block 1420 len 40960 090/300 Read block 2610 len 36864 091/300 Read block 4042 len 20480 092/300 Write block 1544 len 40960 data: 3 2 2 3 4 3 2 3 3 3 093/300 Write block 1685 len 28672 data: 4 4 4 4 2 2 2 094/300 Read block 2846 len 4096 095/300 Write block 1416 len 28672 data: 3 2 2 3 3 4 4 096/300 Write block 9 len 24576 data: 3 5 7 5 5 6 097/300 Write block 1878 len 16384 data: 3 2 3 3 098/300 Read block 3506 len 32768 099/300 Read block 2062 len 8192 100/300 Read block 4381 len 20480 101/300 Write block 4611 len 8192 data: 6 6 102/300 Read block 1512 len 12288 103/300 Write block 311 len 40960 data: 9 8 8 7 6 6 5 6 7 5 104/300 Write block 1077 len 40960 data: 6 5 4 3 3 3 3 4 4 3 105/300 Write block 1550 len 28672 data: 3 4 4 4 3 3 5 106/300 Read block 4024 len 8192 107/300 Read block 3612 len 16384 108/300 Read block 2260 len 28672 109/300 Read block 4052 len 12288 110/300 Write block 4815 len 36864 data: 2 2 2 2 2 3 3 3 4 111/300 Read block 42 len 20480 112/300 Write block 1169 len 4096 data: 3 113/300 Read block 4980 len 12288 114/300 Read block 272 len 28672 115/300 Write block 545 len 40960 data: 2 3 3 3 3 3 3 2 2 2 116/300 Read block 557 len 12288 117/300 Read block 3996 len 32768 118/300 Read block 4015 len 20480 119/300 Read block 308 len 24576 120/300 Write block 1416 len 40960 data: 4 3 3 4 4 5 5 4 4 4 121/300 Read block 450 len 8192 122/300 Write block 2574 len 28672 data: 6 6 6 7 7 7 6 123/300 Write block 825 len 8192 data: 3 3 124/300 Read block 2175 len 12288 125/300 Flush 126/300 Read block 3976 len 12288 127/300 Read block 3329 len 28672 128/300 Read block 2436 len 12288 129/300 Write block 4227 len 4096 data: 4 130/300 Read block 2012 len 16384 131/300 Read block 3438 len 36864 132/300 Read block 4661 len 36864 133/300 Read block 1781 len 28672 134/300 Read block 2978 len 24576 135/300 Read block 1376 len 24576 136/300 Flush 137/300 Write block 1924 len 24576 data: 2 3 3 2 2 2 138/300 Read block 4655 len 16384 139/300 Read block 2986 len 4096 140/300 Read block 439 len 20480 141/300 Flush 142/300 Write block 3109 len 20480 data: 5 4 4 4 5 143/300 Write block 1375 len 24576 data: 3 3 3 3 5 3 144/300 Read block 2863 len 4096 145/300 Read block 1706 len 36864 146/300 Write block 3178 len 28672 data: 5 5 5 4 4 4 4 147/300 Read block 3076 len 12288 148/300 Read block 151 len 24576 149/300 Write block 3454 len 16384 data: 4 5 4 4 150/300 Write block 3391 len 28672 data: 3 2 2 2 2 3 4 151/300 Read block 4024 len 8192 152/300 Read block 2088 len 40960 153/300 Read block 1265 len 12288 154/300 Read block 1789 len 32768 155/300 Read block 2423 len 28672 156/300 Read block 2817 len 36864 157/300 Flush 158/300 Read block 822 len 32768 159/300 Read block 2294 len 32768 160/300 Read block 1758 len 4096 161/300 Read block 4690 len 36864 162/300 Write block 1596 len 8192 data: 5 5 163/300 Read block 3295 len 12288 164/300 Read block 403 len 12288 165/300 Write block 4765 len 24576 data: 4 5 4 4 3 5 166/300 Read block 337 len 36864 167/300 Read block 1650 len 28672 168/300 Write block 929 len 24576 data: 3 3 2 3 3 4 169/300 Write block 262 len 32768 data: 4 3 3 3 3 5 6 5 170/300 Read block 373 len 36864 171/300 Write block 1585 len 4096 data: 5 172/300 Flush 173/300 Write block 1288 len 12288 data: 5 3 3 174/300 Write block 3333 len 24576 data: 5 4 4 4 4 3 175/300 Read block 3987 len 8192 176/300 Read block 3579 len 32768 177/300 Read block 698 len 8192 178/300 Read block 2805 len 28672 179/300 Write block 4497 len 12288 data: 3 5 5 180/300 Read block 4000 len 24576 181/300 Write block 3536 len 36864 data: 4 6 6 6 6 7 7 7 6 182/300 Read block 1623 len 24576 183/300 Write block 2908 len 28672 data: 2 2 2 4 5 6 4 184/300 Flush 185/300 Read block 2131 len 28672 186/300 Read block 4918 len 28672 187/300 Write block 4010 len 36864 data: 7 6 6 6 7 5 5 5 5 188/300 Read block 3324 len 28672 189/300 Read block 821 len 8192 190/300 Read block 3994 len 8192 191/300 Write block 2708 len 28672 data: 2 3 4 3 3 3 4 192/300 Flush 193/300 Read block 3169 len 32768 194/300 Read block 443 len 28672 195/300 Write block 2849 len 12288 data: 5 3 4 196/300 Read block 721 len 24576 197/300 Write block 1027 len 40960 data: 3 3 3 3 2 2 2 2 3 4 198/300 Read block 2843 len 4096 199/300 Write block 2622 len 32768 data: 4 4 4 4 4 6 6 5 200/300 Write block 4608 len 36864 data: 5 6 6 7 7 6 6 5 6 201/300 Read block 3384 len 32768 202/300 Read block 1995 len 8192 203/300 Read block 65 len 24576 204/300 Read block 3927 len 28672 205/300 Write block 3225 len 4096 data: 5 206/300 Write block 1694 len 20480 data: 4 4 4 4 4 207/300 Read block 3571 len 24576 208/300 Read block 753 len 20480 209/300 Write block 1692 len 32768 data: 3 3 5 5 5 5 5 3 210/300 Write block 1874 len 8192 data: 3 3 211/300 Read block 4793 len 40960 212/300 Read block 4199 len 32768 213/300 Read block 2686 len 12288 214/300 Read block 967 len 36864 215/300 Write block 2427 len 20480 data: 2 3 2 2 2 216/300 Write block 2317 len 20480 data: 3 3 3 3 3 217/300 Read block 470 len 36864 218/300 Write block 3246 len 20480 data: 3 4 4 4 3 219/300 Read block 3272 len 24576 220/300 Write block 1895 len 8192 data: 3 3 221/300 Read block 2774 len 40960 222/300 Read block 2903 len 32768 223/300 Write block 2511 len 28672 data: 8 6 6 5 4 4 4 224/300 Write block 4563 len 32768 data: 4 4 4 4 4 4 4 4 225/300 Write block 1129 len 28672 data: 4 4 4 5 5 7 7 226/300 Write block 2685 len 4096 data: 4 227/300 Read block 2078 len 40960 228/300 Write block 1318 len 24576 data: 3 2 2 3 5 5 229/300 Write block 1519 len 36864 data: 3 3 4 4 3 3 3 4 4 230/300 Write block 4646 len 36864 data: 3 3 4 4 4 5 5 5 5 231/300 Write block 563 len 16384 data: 2 2 2 2 232/300 Write block 511 len 24576 data: 2 2 3 4 4 5 233/300 Write block 4589 len 16384 data: 3 3 3 3 234/300 Read block 1900 len 12288 235/300 Read block 3174 len 36864 236/300 Write block 3996 len 12288 data: 7 7 5 237/300 Read block 2596 len 16384 238/300 Write block 75 len 28672 data: 3 3 4 4 4 4 4 239/300 Write block 4677 len 8192 data: 3 3 240/300 Write block 2854 len 32768 data: 4 4 4 5 5 5 4 3 241/300 Flush 242/300 Read block 3041 len 20480 243/300 Read block 4662 len 8192 244/300 Read block 122 len 12288 245/300 Write block 4290 len 24576 data: 3 3 3 4 4 4 246/300 Flush 247/300 Read block 599 len 4096 248/300 Read block 2767 len 36864 249/300 Read block 1734 len 40960 250/300 Read block 1746 len 32768 251/300 Read block 4329 len 40960 252/300 Write block 4326 len 8192 data: 5 5 253/300 Write block 1495 len 24576 data: 6 4 3 3 4 4 254/300 Read block 3156 len 12288 255/300 Write block 4 len 4096 data: 3 256/300 Write block 1675 len 4096 data: 5 257/300 Write block 2298 len 20480 data: 3 3 4 5 5 258/300 Read block 2635 len 36864 259/300 Read block 2554 len 16384 260/300 Read block 4588 len 8192 261/300 Read block 1612 len 4096 262/300 Read block 1319 len 20480 263/300 Write block 4154 len 4096 data: 3 264/300 Flush 265/300 Write block 3798 len 16384 data: 4 4 3 3 266/300 Write block 3025 len 12288 data: 3 3 4 267/300 Write block 3625 len 8192 data: 6 5 268/300 Write block 3927 len 16384 data: 6 5 4 5 269/300 Read block 2194 len 32768 270/300 Read block 2616 len 16384 271/300 Write block 1546 len 32768 data: 3 4 5 4 4 5 5 5 272/300 Read block 3734 len 40960 273/300 Read block 4334 len 4096 274/300 Read block 642 len 32768 275/300 Read block 1469 len 40960 276/300 Read block 1519 len 32768 277/300 Read block 2509 len 28672 278/300 Write block 60 len 8192 data: 4 4 279/300 Write block 3132 len 36864 data: 3 4 5 4 5 5 6 5 5 280/300 Write block 4085 len 40960 data: 4 4 4 4 3 5 5 4 3 4 281/300 Write block 3125 len 20480 data: 4 4 4 3 3 282/300 Write block 2615 len 28672 data: 5 5 4 5 5 5 5 283/300 Read block 3130 len 24576 284/300 Read block 4464 len 20480 285/300 Write block 4043 len 32768 data: 5 4 5 5 6 5 5 3 286/300 Read block 4453 len 32768 287/300 Write block 4877 len 4096 data: 6 288/300 Read block 213 len 16384 289/300 Flush 290/300 Read block 87 len 24576 291/300 Read block 1948 len 24576 292/300 Read block 4125 len 28672 293/300 Read block 435 len 24576 294/300 Read block 2675 len 8192 295/300 Read block 3977 len 32768 296/300 Read block 1685 len 40960 297/300 Flush 298/300 Write block 2514 len 32768 data: 6 5 5 5 4 2 2 2 299/300 Write block 1537 len 4096 data: 2 300/300 Write block 2053 len 16384 data: 4 4 3 4 Main task has work to do, go find it got dsc Start(1) start 1 Jan 02 22:33:14.417 INFO request completed, latency_us: 331, response_code: 204, uri: /start/cid/1, method: POST, req_id: 54d82612-1acf-4692-8e73-c0fae113997b, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 3929 Acked 4928 Write 1 Done New Done true 3930 Acked 4929 Write 4 Done New Done true 3931 Acked 4930 Write 5 Done New Done true 3932 Acked 4931 Flush 0 Done New Done true 3933 Acked 4932 Write 10 Done New Done false 3934 Acked 4933 Read 4 Done New Done false 3935 Acked 4934 Read 6 Done New Done false 3936 Acked 4935 Read 9 Done New Done false 3937 Acked 4936 Flush 0 Done New Done false 3938 Acked 4937 Read 7 Done New Done false 3939 Acked 4938 Flush 0 Done New Done false 3940 Acked 4939 Write 1 Done New Done false 3941 Acked 4940 Write 4 Done New Done false 3942 Acked 4941 Write 6 Done New Done false 3943 Acked 4942 Write 2 Done New Done false 3944 Acked 4943 Read 7 Done New Done false 3945 Acked 4944 Write 6 Done New Done false 3946 Acked 4945 Read 7 Done New Done false 3947 Acked 4946 Read 2 Done New Done false 3948 Acked 4947 Read 7 Done New Done false 3949 Acked 4948 Write 8 Done New Done false 3950 Acked 4949 Read 9 Done New Done false 3951 Acked 4950 Write 2 Done New Done false 3952 Acked 4951 Flush 0 Done New Done false 3953 Acked 4952 Write 3 Done New Done false 3954 Acked 4953 Write 10 Done New Done false 3955 Acked 4954 Read 1 Done New Done false 3956 Acked 4955 Read 3 Done New Done false 3957 Acked 4956 Flush 0 Done New Done false 3958 Acked 4957 Read 7 Done New Done false 3959 Acked 4958 Write 6 Done New Done false 3960 Acked 4959 Write 10 Done New Done false 3961 Acked 4960 Flush 0 Done New Done false 3962 Acked 4961 Write 9 Done New Done false 3963 Acked 4962 Read 8 Done New Done false 3964 Acked 4963 Read 5 Done New Done false 3965 Acked 4964 Write 7 Done New Done false 3966 Acked 4965 Read 4 Done New Done false 3967 Acked 4966 Read 10 Done New Done false 3968 Acked 4967 Read 3 Done New Done false 3969 Acked 4968 Read 5 Done New Done false 3970 Acked 4969 Write 9 Done New Done false 3971 Acked 4970 Read 6 Done New Done false 3972 Acked 4971 Read 6 Done New Done false 3973 Acked 4972 Read 2 Done New Done false 3974 Acked 4973 Read 10 Done New Done false 3975 Acked 4974 Read 1 Done New Done false 3976 Acked 4975 Read 4 Done New Done false 3977 Acked 4976 Flush 0 Done New Done false 3978 Acked 4977 Write 5 Done New Done false 3979 Acked 4978 Read 9 Done New Done false 3980 Acked 4979 Read 9 Done New Done false 3981 Acked 4980 Write 3 Done New Done false 3982 Acked 4981 Write 6 Done New Done false 3983 Acked 4982 Read 6 Done New Done false 3984 Acked 4983 Read 10 Done New Done false 3985 Acked 4984 Write 2 Done New Done false 3986 Acked 4985 Flush 0 Done New Done false 3987 Acked 4986 Write 8 Done New Done false 3988 Acked 4987 Read 1 Done New Done false 3989 Acked 4988 Write 8 Done New Done false 3990 Acked 4989 Read 4 Done New Done false 3991 Acked 4990 Write 6 Done New Done false 3992 Acked 4991 Flush 0 Done New Done false 3993 Acked 4992 Write 5 Done New Done false 3994 Acked 4993 Write 2 Done New Done false 3995 Acked 4994 Read 3 Done New Done false 3996 Acked 4995 Read 6 Done New Done false 3997 Acked 4996 Write 5 Done New Done false 3998 Acked 4997 Write 4 Done New Done false 3999 Acked 4998 Read 9 Done New Done false 4000 Acked 4999 Flush 0 Done New Done false 4001 Acked 5000 Read 3 Done New Done false 4002 Acked 5001 Read 1 Done New Done false 4003 Acked 5002 Write 6 Done New Done false 4004 Acked 5003 Read 3 Done New Done false 4005 Acked 5004 Write 9 Done New Done false 4006 Acked 5005 Write 1 Done New Done false 4007 Acked 5006 Write 10 Done New Done false 4008 Acked 5007 Read 6 Done New Done false 4009 Acked 5008 Read 1 Done New Done false 4010 Acked 5009 Read 1 Done New Done false 4011 Acked 5010 Read 3 Done New Done false 4012 Acked 5011 Read 3 Done New Done false 4013 Acked 5012 Write 1 Done New Done false 4014 Acked 5013 Write 5 Done New Done false 4015 Acked 5014 Read 6 Done New Done false 4016 Acked 5015 Read 3 Done New Done false 4017 Acked 5016 Read 10 Done New Done false 4018 Acked 5017 Read 9 Done New Done false 4019 Acked 5018 Read 5 Done New Done false 4020 Acked 5019 Write 10 Done New Done false 4021 Acked 5020 Write 7 Done New Done false 4022 Acked 5021 Read 1 Done New Done false 4023 Acked 5022 Write 7 Done New Done false 4024 Acked 5023 Write 6 Done New Done false 4025 Acked 5024 Write 4 Done New Done false 4026 Acked 5025 Read 8 Done New Done false 4027 Acked 5026 Read 2 Done New Done false 4028 Acked 5027 Read 5 Done New Done false 4029 Acked 5028 Write 2 Done New Done false 4030 Acked 5029 Read 3 Done New Done false 4031 Acked 5030 Write 10 Done New Done false 4032 Acked 5031 Write 10 Done New Done false 4033 Acked 5032 Write 7 Done New Done false 4034 Acked 5033 Read 2 Done New Done false 4035 Acked 5034 Read 4 Done New Done false 4036 Acked 5035 Read 7 Done New Done false 4037 Acked 5036 Read 3 Done New Done false 4038 Acked 5037 Write 9 Done New Done false 4039 Acked 5038 Read 5 Done New Done false 4040 Acked 5039 Write 1 Done New Done false 4041 Acked 5040 Read 3 Done New Done false 4042 Acked 5041 Read 7 Done New Done false 4043 Acked 5042 Write 10 Done New Done false 4044 Acked 5043 Read 3 Done New Done false 4045 Acked 5044 Read 8 Done New Done false 4046 Acked 5045 Read 5 Done New Done false 4047 Acked 5046 Read 6 Done New Done false 4048 Acked 5047 Write 10 Done New Done false 4049 Acked 5048 Read 2 Done New Done false 4050 Acked 5049 Write 7 Done New Done false 4051 Acked 5050 Write 2 Done New Done false 4052 Acked 5051 Read 3 Done New Done false 4053 Acked 5052 Flush 0 Done New Done false 4054 Acked 5053 Read 3 Done New Done false 4055 Acked 5054 Read 7 Done New Done false 4056 Acked 5055 Read 3 Done New Done false 4057 Acked 5056 Write 1 Done New Done false 4058 Acked 5057 Read 4 Done New Done false 4059 Acked 5058 Read 9 Done New Done false 4060 Acked 5059 Read 9 Done New Done false 4061 Acked 5060 Read 7 Done New Done false 4062 Acked 5061 Read 6 Done New Done false 4063 Acked 5062 Read 6 Done New Done false 4064 Acked 5063 Flush 0 Done New Done false 4065 Acked 5064 Write 6 Done New Done false 4066 Acked 5065 Read 4 Done New Done false 4067 Acked 5066 Read 1 Done New Done false 4068 Acked 5067 Read 5 Done New Done false 4069 Acked 5068 Flush 0 Done New Done false 4070 Acked 5069 Write 5 Done New Done false 4071 Acked 5070 Write 6 Done New Done false 4072 Acked 5071 Read 1 Done New Done false 4073 Acked 5072 Read 9 Done New Done false 4074 Acked 5073 Write 7 Done New Done false 4075 Acked 5074 Read 3 Done New Done false 4076 Acked 5075 Read 6 Done New Done false 4077 Acked 5076 Write 4 Done New Done false 4078 Acked 5077 Write 7 Done New Done false 4079 Acked 5078 Read 2 Done New Done false 4080 Acked 5079 Read 10 Done New Done false 4081 Acked 5080 Read 3 Done New Done false 4082 Acked 5081 Read 8 Done New Done false 4083 Acked 5082 Read 7 Done New Done false 4084 Acked 5083 Read 9 Done New Done false 4085 Acked 5084 Flush 0 Done New Done false 4086 Acked 5085 Read 8 Done New Done false 4087 Acked 5086 Flush 0 Done New Done false 4088 Acked 5087 Read 8 Done New Done false 4089 Acked 5088 Read 1 Done New Done false 4090 Acked 5089 Read 9 Done New Done false 4091 Acked 5090 Write 2 Done New Done false 4092 Acked 5091 Read 3 Done New Done false 4093 Acked 5092 Read 3 Done New Done false 4094 Acked 5093 Write 6 Done New Done false 4095 Acked 5094 Read 9 Done New Done false 4096 Acked 5095 Read 7 Done New Done false 4097 Acked 5096 Write 6 Done New Done false 4098 Acked 5097 Write 8 Done New Done false 4099 Acked 5098 Read 9 Done New Done false 4100 Acked 5099 Write 1 Done New Done false 4101 Acked 5100 Flush 0 Done New Done false 4102 Acked 5101 Write 3 Done New Done false 4103 Acked 5102 Write 6 Done New Done false 4104 Acked 5103 Read 2 Done New Done false 4105 Acked 5104 Read 8 Done New Done false 4106 Acked 5105 Read 2 Done New Done false 4107 Acked 5106 Read 7 Done New Done false 4108 Acked 5107 Write 3 Done New Done false 4109 Acked 5108 Read 6 Done New Done false 4110 Acked 5109 Write 9 Done New Done false 4111 Acked 5110 Read 6 Done New Done false 4112 Acked 5111 Write 7 Done New Done false 4113 Acked 5112 Flush 0 Done New Done false 4114 Acked 5113 Read 7 Done New Done false 4115 Acked 5114 Read 7 Done New Done false 4116 Acked 5115 Write 9 Done New Done false 4117 Acked 5116 Read 7 Done New Done false 4118 Acked 5117 Read 2 Done New Done false 4119 Acked 5118 Read 2 Done New Done false 4120 Acked 5119 Write 7 Done New Done false 4121 Acked 5120 Flush 0 Done New Done false 4122 Acked 5121 Read 8 Done New Done false 4123 Acked 5122 Read 7 Done New Done false 4124 Acked 5123 Write 3 Done New Done false 4125 Acked 5124 Read 6 Done New Done false 4126 Acked 5125 Write 10 Done New Done false 4127 Acked 5126 Read 1 Done New Done false 4128 Acked 5127 Write 8 Done New Done false 4129 Acked 5128 Write 9 Done New Done false 4130 Acked 5129 Read 8 Done New Done false 4131 Acked 5130 Read 2 Done New Done false 4132 Acked 5131 Read 6 Done New Done false 4133 Acked 5132 Read 7 Done New Done false 4134 Acked 5133 Write 1 Done New Done false 4135 Acked 5134 Write 5 Done New Done false 4136 Acked 5135 Read 6 Done New Done false 4137 Acked 5136 Read 5 Done New Done false 4138 Acked 5137 Write 8 Done New Done false 4139 Acked 5138 Write 2 Done New Done false 4140 Acked 5139 Read 10 Done New Done false 4141 Acked 5140 Read 8 Done New Done false 4142 Acked 5141 Read 3 Done New Done false 4143 Acked 5142 Read 9 Done New Done false 4144 Acked 5143 Write 5 Done New Done false 4145 Acked 5144 Write 5 Done New Done false 4146 Acked 5145 Read 9 Done New Done false 4147 Acked 5146 Write 5 Done New Done false 4148 Acked 5147 Read 6 Done New Done false 4149 Acked 5148 Write 2 Done New Done false 4150 Acked 5149 Read 10 Done New Done false 4151 Acked 5150 Read 8 Done New Done false 4152 Acked 5151 Write 7 Done New Done false 4153 Acked 5152 Write 8 Done New Done false 4154 Acked 5153 Write 7 Done New Done false 4155 Acked 5154 Write 1 Done New Done false 4156 Acked 5155 Read 10 Done New Done false 4157 Acked 5156 Write 6 Done New Done false 4158 Acked 5157 Write 9 Done New Done false 4159 Acked 5158 Write 9 Done New Done false 4160 Acked 5159 Write 4 Done New Done false 4161 Acked 5160 Write 6 Done New Done false 4162 Acked 5161 Write 4 Done New Done false 4163 Acked 5162 Read 3 Done New Done false 4164 Acked 5163 Read 9 Done New Done false 4165 Acked 5164 Write 3 Done New Done false 4166 Acked 5165 Read 4 Done New Done false 4167 Acked 5166 Write 7 Done New Done false 4168 Acked 5167 Write 2 Done New Done false 4169 Acked 5168 Write 8 Done New Done false 4170 Acked 5169 Flush 0 Done New Done false 4171 Acked 5170 Read 5 Done New Done false 4172 Acked 5171 Read 2 Done New Done false 4173 Acked 5172 Read 3 Done New Done false 4174 Acked 5173 Write 6 Done New Done false 4175 Acked 5174 Flush 0 Done New Done false 4176 Acked 5175 Read 1 Done New Done false 4177 Acked 5176 Read 9 Done New Done false 4178 Acked 5177 Read 10 Done New Done false 4179 Acked 5178 Read 8 Done New Done false 4180 Acked 5179 Read 10 Done New Done false 4181 Acked 5180 Write 2 Done New Done false 4182 Acked 5181 Write 6 Done New Done false 4183 Acked 5182 Read 3 Done New Done false 4184 Acked 5183 Write 1 Done New Done false 4185 Acked 5184 Write 1 Done New Done false 4186 Acked 5185 Write 5 Done New Done false 4187 Acked 5186 Read 9 Done New Done false 4188 Acked 5187 Read 4 Done New Done false 4189 Acked 5188 Read 2 Done New Done false 4190 Acked 5189 Read 1 Done New Done false 4191 Acked 5190 Read 5 Done New Done false 4192 Acked 5191 Write 1 Done New Done false 4193 Acked 5192 Flush 0 Done New Done false 4194 Acked 5193 Write 4 Done New Done false 4195 Acked 5194 Write 3 Done New Done false 4196 Acked 5195 Write 2 Done New Done false 4197 Acked 5196 Write 4 Done New Done false 4198 Acked 5197 Read 8 Done New Done false 4199 Acked 5198 Read 4 Done New Done false 4200 Acked 5199 Write 8 Done New Done false 4201 Acked 5200 Read 10 Done New Done false 4202 Acked 5201 Read 1 Done New Done false 4203 Acked 5202 Read 8 Done New Done false 4204 Acked 5203 Read 10 Done New Done false 4205 Acked 5204 Read 8 Done New Done false 4206 Acked 5205 Read 7 Done New Done false 4207 Acked 5206 Write 2 Done New Done false 4208 Acked 5207 Write 9 Done New Done false 4209 Acked 5208 Write 10 Done New Done false 4210 Acked 5209 Write 5 Done New Done false 4211 Acked 5210 Write 7 Done New Done false 4212 Acked 5211 Read 6 Done New Done false 4213 Acked 5212 Read 5 Done New Done false 4214 Acked 5213 Write 8 Done New Done false 4215 Acked 5214 Read 8 Done New Done false 4216 Acked 5215 Write 1 Done New Done false 4217 Acked 5216 Read 4 Done New Done false 4218 Acked 5217 Flush 0 Done New Done false 4219 Acked 5218 Read 6 Done New Done false 4220 Acked 5219 Read 6 Done New Done false 4221 Acked 5220 Read 7 Done New Done false 4222 Acked 5221 Read 6 Done New Done false 4223 Acked 5222 Read 2 Done New Done false 4224 Acked 5223 Read 8 Done New Done false 4225 Acked 5224 Read 10 Done New Done false 4226 Acked 5225 Flush 0 Done New Done false 4227 Acked 5226 Write 8 Done New Done false 4228 Acked 5227 Write 1 Done New Done false 4229 Acked 5228 Write 4 Done New Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 301 0 301 Sent 0 0 0 0 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5225 4927 5225 Downstairs last five completed: 4927 4926 4925 4924 4923 Upstairs last five completed: 4229 4228 4227 4226 4225 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:14.797459925Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:14.79752949Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:14.797558369Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:14.797585143Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:14.797605683Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:14.79762829Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:14.797860515Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:15.798669393Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:15.79874385Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:15.798785651Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:15.798814532Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:15.798835167Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:15.798857819Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:15.799095501Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:16.799734693Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:16.799812337Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:16.799840927Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:16.799865338Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:16.799891722Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:16.799912174Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:16.799945776Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:16.800174548Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:17.802064629Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:17.802131358Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:17.80215917Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:17.802186122Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:17.80220671Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:17.802248026Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:17.802486589Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 3929 Acked 4928 Write 1 Done New Done true 3930 Acked 4929 Write 4 Done New Done true 3931 Acked 4930 Write 5 Done New Done true 3932 Acked 4931 Flush 0 Done New Done true 3933 Acked 4932 Write 10 Done New Done true 3934 Acked 4933 Read 4 Done New Done true 3935 Acked 4934 Read 6 Done New Done true 3936 Acked 4935 Read 9 Done New Done true 3937 Acked 4936 Flush 0 Done New Done true 3938 Acked 4937 Read 7 Done New Done true 3939 Acked 4938 Flush 0 Done New Done true 3940 Acked 4939 Write 1 Done New Done true 3941 Acked 4940 Write 4 Done New Done true 3942 Acked 4941 Write 6 Done New Done true 3943 Acked 4942 Write 2 Done New Done true 3944 Acked 4943 Read 7 Done New Done true 3945 Acked 4944 Write 6 Done New Done true 3946 Acked 4945 Read 7 Done New Done true 3947 Acked 4946 Read 2 Done New Done true 3948 Acked 4947 Read 7 Done New Done true 3949 Acked 4948 Write 8 Done New Done true 3950 Acked 4949 Read 9 Done New Done true 3951 Acked 4950 Write 2 Done New Done true 3952 Acked 4951 Flush 0 Done New Done true 3953 Acked 4952 Write 3 Done New Done true 3954 Acked 4953 Write 10 Done New Done true 3955 Acked 4954 Read 1 Done New Done true 3956 Acked 4955 Read 3 Done New Done true 3957 Acked 4956 Flush 0 Done New Done true 3958 Acked 4957 Read 7 Done New Done true 3959 Acked 4958 Write 6 Done New Done true 3960 Acked 4959 Write 10 Done New Done true 3961 Acked 4960 Flush 0 Done New Done true 3962 Acked 4961 Write 9 Done New Done true 3963 Acked 4962 Read 8 Done New Done true 3964 Acked 4963 Read 5 Done New Done true 3965 Acked 4964 Write 7 Done New Done true 3966 Acked 4965 Read 4 Done New Done true 3967 Acked 4966 Read 10 Done New Done true 3968 Acked 4967 Read 3 Done New Done true 3969 Acked 4968 Read 5 Done New Done true 3970 Acked 4969 Write 9 Done New Done true 3971 Acked 4970 Read 6 Done New Done true 3972 Acked 4971 Read 6 Done New Done true 3973 Acked 4972 Read 2 Done New Done true 3974 Acked 4973 Read 10 Done New Done true 3975 Acked 4974 Read 1 Done New Done true 3976 Acked 4975 Read 4 Done New Done true 3977 Acked 4976 Flush 0 Done New Done true 3978 Acked 4977 Write 5 Done New Done true 3979 Acked 4978 Read 9 Done New Done true 3980 Acked 4979 Read 9 Done New Done true 3981 Acked 4980 Write 3 Done New Done true 3982 Acked 4981 Write 6 Done New Done true 3983 Acked 4982 Read 6 Done New Done true 3984 Acked 4983 Read 10 Done New Done true 3985 Acked 4984 Write 2 Done New Done true 3986 Acked 4985 Flush 0 Done New Done true 3987 Acked 4986 Write 8 Done New Done true 3988 Acked 4987 Read 1 Done New Done true 3989 Acked 4988 Write 8 Done New Done true 3990 Acked 4989 Read 4 Done New Done true 3991 Acked 4990 Write 6 Done New Done true 3992 Acked 4991 Flush 0 Done New Done true 3993 Acked 4992 Write 5 Done New Done true 3994 Acked 4993 Write 2 Done New Done true 3995 Acked 4994 Read 3 Done New Done true 3996 Acked 4995 Read 6 Done New Done true 3997 Acked 4996 Write 5 Done New Done true 3998 Acked 4997 Write 4 Done New Done true 3999 Acked 4998 Read 9 Done New Done true 4000 Acked 4999 Flush 0 Done New Done true 4001 Acked 5000 Read 3 Done New Done true 4002 Acked 5001 Read 1 Done New Done true 4003 Acked 5002 Write 6 Done New Done true 4004 Acked 5003 Read 3 Done New Done true 4005 Acked 5004 Write 9 Done New Done true 4006 Acked 5005 Write 1 Done New Done true 4007 Acked 5006 Write 10 Done New Done true 4008 Acked 5007 Read 6 Done New Done true 4009 Acked 5008 Read 1 Done New Done true 4010 Acked 5009 Read 1 Done New Done true 4011 Acked 5010 Read 3 Done New Done true 4012 Acked 5011 Read 3 Done New Done true 4013 Acked 5012 Write 1 Done New Done true 4014 Acked 5013 Write 5 Done New Done true 4015 Acked 5014 Read 6 Done New Done true 4016 Acked 5015 Read 3 Done New Done true 4017 Acked 5016 Read 10 Done New Done true 4018 Acked 5017 Read 9 Done New Done true 4019 Acked 5018 Read 5 Done New Done true 4020 Acked 5019 Write 10 Done New Done true 4021 Acked 5020 Write 7 Done New Done true 4022 Acked 5021 Read 1 Done New Done true 4023 Acked 5022 Write 7 Done New Done true 4024 Acked 5023 Write 6 Done New Done true 4025 Acked 5024 Write 4 Done New Done true 4026 Acked 5025 Read 8 Done New Done true 4027 Acked 5026 Read 2 Done New Done true 4028 Acked 5027 Read 5 Done New Done true 4029 Acked 5028 Write 2 Done New Done true 4030 Acked 5029 Read 3 Done New Done true 4031 Acked 5030 Write 10 Done New Done true 4032 Acked 5031 Write 10 Done New Done true 4033 Acked 5032 Write 7 Done New Done true 4034 Acked 5033 Read 2 Done New Done true 4035 Acked 5034 Read 4 Done New Done true 4036 Acked 5035 Read 7 Done New Done true 4037 Acked 5036 Read 3 Done New Done true 4038 Acked 5037 Write 9 Done New Done true 4039 Acked 5038 Read 5 Done New Done true 4040 Acked 5039 Write 1 Done New Done true 4041 Acked 5040 Read 3 Done New Done true 4042 Acked 5041 Read 7 Done New Done true 4043 Acked 5042 Write 10 Done New Done true 4044 Acked 5043 Read 3 Done New Done true 4045 Acked 5044 Read 8 Done New Done true 4046 Acked 5045 Read 5 Done New Done true 4047 Acked 5046 Read 6 Done New Done true 4048 Acked 5047 Write 10 Done New Done true 4049 Acked 5048 Read 2 Done New Done true 4050 Acked 5049 Write 7 Done New Done true 4051 Acked 5050 Write 2 Done New Done true 4052 Acked 5051 Read 3 Done New Done true 4053 Acked 5052 Flush 0 Done New Done true 4054 Acked 5053 Read 3 Done New Done true 4055 Acked 5054 Read 7 Done New Done true 4056 Acked 5055 Read 3 Done New Done true 4057 Acked 5056 Write 1 Done New Done true 4058 Acked 5057 Read 4 Done New Done true 4059 Acked 5058 Read 9 Done New Done true 4060 Acked 5059 Read 9 Done New Done true 4061 Acked 5060 Read 7 Done New Done true 4062 Acked 5061 Read 6 Done New Done true 4063 Acked 5062 Read 6 Done New Done true 4064 Acked 5063 Flush 0 Done New Done true 4065 Acked 5064 Write 6 Done New Done true 4066 Acked 5065 Read 4 Done New Done true 4067 Acked 5066 Read 1 Done New Done true 4068 Acked 5067 Read 5 Done New Done true 4069 Acked 5068 Flush 0 Done New Done true 4070 Acked 5069 Write 5 Done New Done true 4071 Acked 5070 Write 6 Done New Done true 4072 Acked 5071 Read 1 Done New Done true 4073 Acked 5072 Read 9 Done New Done true 4074 Acked 5073 Write 7 Done New Done true 4075 Acked 5074 Read 3 Done New Done true 4076 Acked 5075 Read 6 Done New Done true 4077 Acked 5076 Write 4 Done New Done true 4078 Acked 5077 Write 7 Done New Done true 4079 Acked 5078 Read 2 Done New Done true 4080 Acked 5079 Read 10 Done New Done true 4081 Acked 5080 Read 3 Done New Done true 4082 Acked 5081 Read 8 Done New Done true 4083 Acked 5082 Read 7 Done New Done true 4084 Acked 5083 Read 9 Done New Done true 4085 Acked 5084 Flush 0 Done New Done true 4086 Acked 5085 Read 8 Done New Done true 4087 Acked 5086 Flush 0 Done New Done true 4088 Acked 5087 Read 8 Done New Done true 4089 Acked 5088 Read 1 Done New Done true 4090 Acked 5089 Read 9 Done New Done true 4091 Acked 5090 Write 2 Done New Done true 4092 Acked 5091 Read 3 Done New Done true 4093 Acked 5092 Read 3 Done New Done true 4094 Acked 5093 Write 6 Done New Done true 4095 Acked 5094 Read 9 Done New Done true 4096 Acked 5095 Read 7 Done New Done true 4097 Acked 5096 Write 6 Done New Done true 4098 Acked 5097 Write 8 Done New Done true 4099 Acked 5098 Read 9 Done New Done true 4100 Acked 5099 Write 1 Done New Done true 4101 Acked 5100 Flush 0 Done New Done true 4102 Acked 5101 Write 3 Done New Done true 4103 Acked 5102 Write 6 Done New Done true 4104 Acked 5103 Read 2 Done New Done true 4105 Acked 5104 Read 8 Done New Done true 4106 Acked 5105 Read 2 Done New Done true 4107 Acked 5106 Read 7 Done New Done true 4108 Acked 5107 Write 3 Done New Done true 4109 Acked 5108 Read 6 Done New Done true 4110 Acked 5109 Write 9 Done New Done true 4111 Acked 5110 Read 6 Done New Done true 4112 Acked 5111 Write 7 Done New Done true 4113 Acked 5112 Flush 0 Done New Done true 4114 Acked 5113 Read 7 Done New Done true 4115 Acked 5114 Read 7 Done New Done true 4116 Acked 5115 Write 9 Done New Done true 4117 Acked 5116 Read 7 Done New Done true 4118 Acked 5117 Read 2 Done New Done true 4119 Acked 5118 Read 2 Done New Done true 4120 Acked 5119 Write 7 Done New Done true 4121 Acked 5120 Flush 0 Done New Done true 4122 Acked 5121 Read 8 Done New Done true 4123 Acked 5122 Read 7 Done New Done true 4124 Acked 5123 Write 3 Done New Done true 4125 Acked 5124 Read 6 Done New Done true 4126 Acked 5125 Write 10 Done New Done true 4127 Acked 5126 Read 1 Done New Done true 4128 Acked 5127 Write 8 Done New Done true 4129 Acked 5128 Write 9 Done New Done true 4130 Acked 5129 Read 8 Done New Done true 4131 Acked 5130 Read 2 Done New Done true 4132 Acked 5131 Read 6 Done New Done true 4133 Acked 5132 Read 7 Done New Done true 4134 Acked 5133 Write 1 Done New Done true 4135 Acked 5134 Write 5 Done New Done true 4136 Acked 5135 Read 6 Done New Done true 4137 Acked 5136 Read 5 Done New Done true 4138 Acked 5137 Write 8 Done New Done true 4139 Acked 5138 Write 2 Done New Done true 4140 Acked 5139 Read 10 Done New Done true 4141 Acked 5140 Read 8 Done New Done true 4142 Acked 5141 Read 3 Done New Done true 4143 Acked 5142 Read 9 Done New Done true 4144 Acked 5143 Write 5 Done New Done true 4145 Acked 5144 Write 5 Done New Done true 4146 Acked 5145 Read 9 Done New Done true 4147 Acked 5146 Write 5 Done New Done true 4148 Acked 5147 Read 6 Done New Done true 4149 Acked 5148 Write 2 Done New Done true 4150 Acked 5149 Read 10 Done New Done true 4151 Acked 5150 Read 8 Done New Done true 4152 Acked 5151 Write 7 Done New Done true 4153 Acked 5152 Write 8 Done New Done true 4154 Acked 5153 Write 7 Done New Done true 4155 Acked 5154 Write 1 Done New Done true 4156 Acked 5155 Read 10 Done New Done true 4157 Acked 5156 Write 6 Done New Done true 4158 Acked 5157 Write 9 Done New Done true 4159 Acked 5158 Write 9 Done New Done true 4160 Acked 5159 Write 4 Done New Done true 4161 Acked 5160 Write 6 Done New Done true 4162 Acked 5161 Write 4 Done New Done true 4163 Acked 5162 Read 3 Done New Done true 4164 Acked 5163 Read 9 Done New Done true 4165 Acked 5164 Write 3 Done New Done true 4166 Acked 5165 Read 4 Done New Done true 4167 Acked 5166 Write 7 Done New Done true 4168 Acked 5167 Write 2 Done New Done true 4169 Acked 5168 Write 8 Done New Done true 4170 Acked 5169 Flush 0 Done New Done true 4171 Acked 5170 Read 5 Done New Done true 4172 Acked 5171 Read 2 Done New Done true 4173 Acked 5172 Read 3 Done New Done true 4174 Acked 5173 Write 6 Done New Done true 4175 Acked 5174 Flush 0 Done New Done true 4176 Acked 5175 Read 1 Done New Done true 4177 Acked 5176 Read 9 Done New Done true 4178 Acked 5177 Read 10 Done New Done true 4179 Acked 5178 Read 8 Done New Done true 4180 Acked 5179 Read 10 Done New Done true 4181 Acked 5180 Write 2 Done New Done true 4182 Acked 5181 Write 6 Done New Done true 4183 Acked 5182 Read 3 Done New Done true 4184 Acked 5183 Write 1 Done New Done true 4185 Acked 5184 Write 1 Done New Done true 4186 Acked 5185 Write 5 Done New Done true 4187 Acked 5186 Read 9 Done New Done true 4188 Acked 5187 Read 4 Done New Done true 4189 Acked 5188 Read 2 Done New Done true 4190 Acked 5189 Read 1 Done New Done true 4191 Acked 5190 Read 5 Done New Done true 4192 Acked 5191 Write 1 Done New Done true 4193 Acked 5192 Flush 0 Done New Done true 4194 Acked 5193 Write 4 Done New Done true 4195 Acked 5194 Write 3 Done New Done true 4196 Acked 5195 Write 2 Done New Done true 4197 Acked 5196 Write 4 Done New Done true 4198 Acked 5197 Read 8 Done New Done true 4199 Acked 5198 Read 4 Done New Done true 4200 Acked 5199 Write 8 Done New Done true 4201 Acked 5200 Read 10 Done New Done true 4202 Acked 5201 Read 1 Done New Done true 4203 Acked 5202 Read 8 Done New Done true 4204 Acked 5203 Read 10 Done New Done true 4205 Acked 5204 Read 8 Done New Done true 4206 Acked 5205 Read 7 Done New Done true 4207 Acked 5206 Write 2 Done New Done true 4208 Acked 5207 Write 9 Done New Done true 4209 Acked 5208 Write 10 Done New Done true 4210 Acked 5209 Write 5 Done New Done true 4211 Acked 5210 Write 7 Done New Done true 4212 Acked 5211 Read 6 Done New Done true 4213 Acked 5212 Read 5 Done New Done true 4214 Acked 5213 Write 8 Done New Done true 4215 Acked 5214 Read 8 Done New Done true 4216 Acked 5215 Write 1 Done New Done true 4217 Acked 5216 Read 4 Done New Done true 4218 Acked 5217 Flush 0 Done New Done true 4219 Acked 5218 Read 6 Done New Done true 4220 Acked 5219 Read 6 Done New Done true 4221 Acked 5220 Read 7 Done New Done true 4222 Acked 5221 Read 6 Done New Done true 4223 Acked 5222 Read 2 Done New Done true 4224 Acked 5223 Read 8 Done New Done true 4225 Acked 5224 Read 10 Done New Done true 4226 Acked 5225 Flush 0 Done New Done true 4227 Acked 5226 Write 8 Done New Done true 4228 Acked 5227 Write 1 Done New Done true 4229 Acked 5228 Write 4 Done New Done true 4230 Acked 5229 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 302 0 302 Sent 0 0 0 0 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5229 4927 5229 Downstairs last five completed: 4927 4926 4925 4924 4923 Upstairs last five completed: 4230 4229 4228 4227 4226 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:18.803481645Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:18.803547609Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:18.803575298Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:18.803601963Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:18.803622671Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:18.803655604Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:18.803894935Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:19.80465437Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:19.804734308Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:19.804777842Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:19.804806656Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:19.804827207Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:19.804849941Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:19.805085931Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:20.805875909Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:20.805943075Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:20.805970752Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:20.805997426Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:20.806017841Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:20.806048621Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:20.806289783Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:21.807266377Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:21.807330733Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:21.807358219Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:21.807382412Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:21.80741912Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:21.807456568Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1199) {"msg":"[1] client re-new 302 jobs since flush 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:21.807495375Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820][1] reports Running {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:21.807751181Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 3929 Acked 4928 Write 1 Done New Done true 3930 Acked 4929 Write 4 Done New Done true 3931 Acked 4930 Write 5 Done New Done true 3932 Acked 4931 Flush 0 Done New Done true 3933 Acked 4932 Write 10 Done New Done true 3934 Acked 4933 Read 4 Done New Done true 3935 Acked 4934 Read 6 Done New Done true 3936 Acked 4935 Read 9 Done New Done true 3937 Acked 4936 Flush 0 Done New Done true 3938 Acked 4937 Read 7 Done New Done true 3939 Acked 4938 Flush 0 Done New Done true 3940 Acked 4939 Write 1 Done New Done true 3941 Acked 4940 Write 4 Done New Done true 3942 Acked 4941 Write 6 Done New Done true 3943 Acked 4942 Write 2 Done New Done true 3944 Acked 4943 Read 7 Done New Done true 3945 Acked 4944 Write 6 Done New Done true 3946 Acked 4945 Read 7 Done New Done true 3947 Acked 4946 Read 2 Done New Done true 3948 Acked 4947 Read 7 Done New Done true 3949 Acked 4948 Write 8 Done New Done true 3950 Acked 4949 Read 9 Done New Done true 3951 Acked 4950 Write 2 Done New Done true 3952 Acked 4951 Flush 0 Done New Done true 3953 Acked 4952 Write 3 Done New Done true 3954 Acked 4953 Write 10 Done New Done true 3955 Acked 4954 Read 1 Done New Done true 3956 Acked 4955 Read 3 Done New Done true 3957 Acked 4956 Flush 0 Done New Done true 3958 Acked 4957 Read 7 Done New Done true 3959 Acked 4958 Write 6 Done New Done true 3960 Acked 4959 Write 10 Done New Done true 3961 Acked 4960 Flush 0 Done New Done true 3962 Acked 4961 Write 9 Done New Done true 3963 Acked 4962 Read 8 Done New Done true 3964 Acked 4963 Read 5 Done New Done true 3965 Acked 4964 Write 7 Done New Done true 3966 Acked 4965 Read 4 Done New Done true 3967 Acked 4966 Read 10 Done New Done true 3968 Acked 4967 Read 3 Done New Done true 3969 Acked 4968 Read 5 Done New Done true 3970 Acked 4969 Write 9 Done New Done true 3971 Acked 4970 Read 6 Done New Done true 3972 Acked 4971 Read 6 Done New Done true 3973 Acked 4972 Read 2 Done New Done true 3974 Acked 4973 Read 10 Done New Done true 3975 Acked 4974 Read 1 Done New Done true 3976 Acked 4975 Read 4 Done New Done true 3977 Acked 4976 Flush 0 Done New Done true 3978 Acked 4977 Write 5 Done New Done true 3979 Acked 4978 Read 9 Done New Done true 3980 Acked 4979 Read 9 Done New Done true 3981 Acked 4980 Write 3 Done New Done true 3982 Acked 4981 Write 6 Done New Done true 3983 Acked 4982 Read 6 Done New Done true 3984 Acked 4983 Read 10 Done New Done true 3985 Acked 4984 Write 2 Done New Done true 3986 Acked 4985 Flush 0 Done New Done true 3987 Acked 4986 Write 8 Done New Done true 3988 Acked 4987 Read 1 Done New Done true 3989 Acked 4988 Write 8 Done New Done true 3990 Acked 4989 Read 4 Done New Done true 3991 Acked 4990 Write 6 Done New Done true 3992 Acked 4991 Flush 0 Done New Done true 3993 Acked 4992 Write 5 Done New Done true 3994 Acked 4993 Write 2 Done New Done true 3995 Acked 4994 Read 3 Done New Done true 3996 Acked 4995 Read 6 Done New Done true 3997 Acked 4996 Write 5 Done New Done true 3998 Acked 4997 Write 4 Done New Done true 3999 Acked 4998 Read 9 Done New Done true 4000 Acked 4999 Flush 0 Done New Done true 4001 Acked 5000 Read 3 Done New Done true 4002 Acked 5001 Read 1 Done New Done true 4003 Acked 5002 Write 6 Done New Done true 4004 Acked 5003 Read 3 Done New Done true 4005 Acked 5004 Write 9 Done New Done true 4006 Acked 5005 Write 1 Done New Done true 4007 Acked 5006 Write 10 Done New Done true 4008 Acked 5007 Read 6 Done New Done true 4009 Acked 5008 Read 1 Done New Done true 4010 Acked 5009 Read 1 Done New Done true 4011 Acked 5010 Read 3 Done New Done true 4012 Acked 5011 Read 3 Done New Done true 4013 Acked 5012 Write 1 Done New Done true 4014 Acked 5013 Write 5 Done New Done true 4015 Acked 5014 Read 6 Done New Done true 4016 Acked 5015 Read 3 Done New Done true 4017 Acked 5016 Read 10 Done New Done true 4018 Acked 5017 Read 9 Done New Done true 4019 Acked 5018 Read 5 Done New Done true 4020 Acked 5019 Write 10 Done New Done true 4021 Acked 5020 Write 7 Done New Done true 4022 Acked 5021 Read 1 Done New Done true 4023 Acked 5022 Write 7 Done New Done true 4024 Acked 5023 Write 6 Done New Done true 4025 Acked 5024 Write 4 Done New Done true 4026 Acked 5025 Read 8 Done New Done true 4027 Acked 5026 Read 2 Done New Done true 4028 Acked 5027 Read 5 Done New Done true 4029 Acked 5028 Write 2 Done New Done true 4030 Acked 5029 Read 3 Done New Done true 4031 Acked 5030 Write 10 Done New Done true 4032 Acked 5031 Write 10 Done New Done true 4033 Acked 5032 Write 7 Done New Done true 4034 Acked 5033 Read 2 Done New Done true 4035 Acked 5034 Read 4 Done New Done true 4036 Acked 5035 Read 7 Done New Done true 4037 Acked 5036 Read 3 Done New Done true 4038 Acked 5037 Write 9 Done New Done true 4039 Acked 5038 Read 5 Done New Done true 4040 Acked 5039 Write 1 Done New Done true 4041 Acked 5040 Read 3 Done New Done true 4042 Acked 5041 Read 7 Done New Done true 4043 Acked 5042 Write 10 Done New Done true 4044 Acked 5043 Read 3 Done New Done true 4045 Acked 5044 Read 8 Done New Done true 4046 Acked 5045 Read 5 Done New Done true 4047 Acked 5046 Read 6 Done New Done true 4048 Acked 5047 Write 10 Done New Done true 4049 Acked 5048 Read 2 Done New Done true 4050 Acked 5049 Write 7 Done New Done true 4051 Acked 5050 Write 2 Done New Done true 4052 Acked 5051 Read 3 Done New Done true 4053 Acked 5052 Flush 0 Done New Done true 4054 Acked 5053 Read 3 Done New Done true 4055 Acked 5054 Read 7 Done New Done true 4056 Acked 5055 Read 3 Done New Done true 4057 Acked 5056 Write 1 Done New Done true 4058 Acked 5057 Read 4 Done New Done true 4059 Acked 5058 Read 9 Done New Done true 4060 Acked 5059 Read 9 Done New Done true 4061 Acked 5060 Read 7 Done New Done true 4062 Acked 5061 Read 6 Done New Done true 4063 Acked 5062 Read 6 Done New Done true 4064 Acked 5063 Flush 0 Done New Done true 4065 Acked 5064 Write 6 Done New Done true 4066 Acked 5065 Read 4 Done New Done true 4067 Acked 5066 Read 1 Done New Done true 4068 Acked 5067 Read 5 Done New Done true 4069 Acked 5068 Flush 0 Done New Done true 4070 Acked 5069 Write 5 Done New Done true 4071 Acked 5070 Write 6 Done New Done true 4072 Acked 5071 Read 1 Done New Done true 4073 Acked 5072 Read 9 Done New Done true 4074 Acked 5073 Write 7 Done New Done true 4075 Acked 5074 Read 3 Done New Done true 4076 Acked 5075 Read 6 Done New Done true 4077 Acked 5076 Write 4 Done New Done true 4078 Acked 5077 Write 7 Done New Done true 4079 Acked 5078 Read 2 Done New Done true 4080 Acked 5079 Read 10 Done New Done true 4081 Acked 5080 Read 3 Done New Done true 4082 Acked 5081 Read 8 Done New Done true 4083 Acked 5082 Read 7 Done New Done true 4084 Acked 5083 Read 9 Done New Done true 4085 Acked 5084 Flush 0 Done New Done true 4086 Acked 5085 Read 8 Done New Done true 4087 Acked 5086 Flush 0 Done New Done true 4088 Acked 5087 Read 8 Done New Done true 4089 Acked 5088 Read 1 Done New Done true 4090 Acked 5089 Read 9 Done New Done true 4091 Acked 5090 Write 2 Done New Done true 4092 Acked 5091 Read 3 Done New Done true 4093 Acked 5092 Read 3 Done New Done true 4094 Acked 5093 Write 6 Done New Done true 4095 Acked 5094 Read 9 Done New Done true 4096 Acked 5095 Read 7 Done New Done true 4097 Acked 5096 Write 6 Done New Done true 4098 Acked 5097 Write 8 Done New Done true 4099 Acked 5098 Read 9 Done New Done true 4100 Acked 5099 Write 1 Done New Done true 4101 Acked 5100 Flush 0 Done New Done true 4102 Acked 5101 Write 3 Done New Done true 4103 Acked 5102 Write 6 Done New Done true 4104 Acked 5103 Read 2 Done New Done true 4105 Acked 5104 Read 8 Done New Done true 4106 Acked 5105 Read 2 Done New Done true 4107 Acked 5106 Read 7 Done New Done true 4108 Acked 5107 Write 3 Done New Done true 4109 Acked 5108 Read 6 Done New Done true 4110 Acked 5109 Write 9 Done New Done true 4111 Acked 5110 Read 6 Done New Done true 4112 Acked 5111 Write 7 Done New Done true 4113 Acked 5112 Flush 0 Done New Done true 4114 Acked 5113 Read 7 Done New Done true 4115 Acked 5114 Read 7 Done New Done true 4116 Acked 5115 Write 9 Done New Done true 4117 Acked 5116 Read 7 Done New Done true 4118 Acked 5117 Read 2 Done New Done true 4119 Acked 5118 Read 2 Done New Done true 4120 Acked 5119 Write 7 Done New Done true 4121 Acked 5120 Flush 0 Done New Done true 4122 Acked 5121 Read 8 Done New Done true 4123 Acked 5122 Read 7 Done New Done true 4124 Acked 5123 Write 3 Done New Done true 4125 Acked 5124 Read 6 Done New Done true 4126 Acked 5125 Write 10 Done New Done true 4127 Acked 5126 Read 1 Done New Done true 4128 Acked 5127 Write 8 Done New Done true 4129 Acked 5128 Write 9 Done New Done true 4130 Acked 5129 Read 8 Done New Done true 4131 Acked 5130 Read 2 Done New Done true 4132 Acked 5131 Read 6 Done New Done true 4133 Acked 5132 Read 7 Done New Done true 4134 Acked 5133 Write 1 Done New Done true 4135 Acked 5134 Write 5 Done New Done true 4136 Acked 5135 Read 6 Done New Done true 4137 Acked 5136 Read 5 Done New Done true 4138 Acked 5137 Write 8 Done New Done true 4139 Acked 5138 Write 2 Done New Done true 4140 Acked 5139 Read 10 Done New Done true 4141 Acked 5140 Read 8 Done New Done true 4142 Acked 5141 Read 3 Done New Done true 4143 Acked 5142 Read 9 Done New Done true 4144 Acked 5143 Write 5 Done New Done true 4145 Acked 5144 Write 5 Done New Done true 4146 Acked 5145 Read 9 Done New Done true 4147 Acked 5146 Write 5 Done New Done true 4148 Acked 5147 Read 6 Done New Done true 4149 Acked 5148 Write 2 Done New Done true 4150 Acked 5149 Read 10 Done New Done true 4151 Acked 5150 Read 8 Done New Done true 4152 Acked 5151 Write 7 Done New Done true 4153 Acked 5152 Write 8 Done New Done true 4154 Acked 5153 Write 7 Done New Done true 4155 Acked 5154 Write 1 Done New Done true 4156 Acked 5155 Read 10 Done New Done true 4157 Acked 5156 Write 6 Done New Done true 4158 Acked 5157 Write 9 Done New Done true 4159 Acked 5158 Write 9 Done New Done true 4160 Acked 5159 Write 4 Done New Done true 4161 Acked 5160 Write 6 Done New Done true 4162 Acked 5161 Write 4 Done New Done true 4163 Acked 5162 Read 3 Done New Done true 4164 Acked 5163 Read 9 Done New Done true 4165 Acked 5164 Write 3 Done New Done true 4166 Acked 5165 Read 4 Done New Done true 4167 Acked 5166 Write 7 Done New Done true 4168 Acked 5167 Write 2 Done New Done true 4169 Acked 5168 Write 8 Done New Done true 4170 Acked 5169 Flush 0 Done New Done true 4171 Acked 5170 Read 5 Done New Done true 4172 Acked 5171 Read 2 Done New Done true 4173 Acked 5172 Read 3 Done New Done true 4174 Acked 5173 Write 6 Done New Done true 4175 Acked 5174 Flush 0 Done New Done true 4176 Acked 5175 Read 1 Done New Done true 4177 Acked 5176 Read 9 Done New Done true 4178 Acked 5177 Read 10 Done New Done true 4179 Acked 5178 Read 8 Done New Done true 4180 Acked 5179 Read 10 Done New Done true 4181 Acked 5180 Write 2 Done New Done true 4182 Acked 5181 Write 6 Done New Done true 4183 Acked 5182 Read 3 Done New Done true 4184 Acked 5183 Write 1 Done New Done true 4185 Acked 5184 Write 1 Done New Done true 4186 Acked 5185 Write 5 Done New Done true 4187 Acked 5186 Read 9 Done New Done true 4188 Acked 5187 Read 4 Done New Done true 4189 Acked 5188 Read 2 Done New Done true 4190 Acked 5189 Read 1 Done New Done true 4191 Acked 5190 Read 5 Done New Done true 4192 Acked 5191 Write 1 Done New Done true 4193 Acked 5192 Flush 0 Done New Done true 4194 Acked 5193 Write 4 Done New Done true 4195 Acked 5194 Write 3 Done New Done true 4196 Acked 5195 Write 2 Done New Done true 4197 Acked 5196 Write 4 Done New Done true 4198 Acked 5197 Read 8 Done New Done true 4199 Acked 5198 Read 4 Done New Done true 4200 Acked 5199 Write 8 Done New Done true 4201 Acked 5200 Read 10 Done New Done true 4202 Acked 5201 Read 1 Done New Done true 4203 Acked 5202 Read 8 Done New Done true 4204 Acked 5203 Read 10 Done New Done true 4205 Acked 5204 Read 8 Done New Done true 4206 Acked 5205 Read 7 Done New Done true 4207 Acked 5206 Write 2 Done New Done true 4208 Acked 5207 Write 9 Done New Done true 4209 Acked 5208 Write 10 Done New Done true 4210 Acked 5209 Write 5 Done New Done true 4211 Acked 5210 Write 7 Done New Done true 4212 Acked 5211 Read 6 Done New Done true 4213 Acked 5212 Read 5 Done New Done true 4214 Acked 5213 Write 8 Done New Done true 4215 Acked 5214 Read 8 Done New Done true 4216 Acked 5215 Write 1 Done New Done true 4217 Acked 5216 Read 4 Done New Done true 4218 Acked 5217 Flush 0 Done New Done true 4219 Acked 5218 Read 6 Done New Done true 4220 Acked 5219 Read 6 Done New Done true 4221 Acked 5220 Read 7 Done New Done true 4222 Acked 5221 Read 6 Done New Done true 4223 Acked 5222 Read 2 Done New Done true 4224 Acked 5223 Read 8 Done New Done true 4225 Acked 5224 Read 10 Done New Done true 4226 Acked 5225 Flush 0 Done New Done true 4227 Acked 5226 Write 8 Done New Done true 4228 Acked 5227 Write 1 Done New Done true 4229 Acked 5228 Write 4 Done New Done true 4230 Acked 5229 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 302 0 302 Sent 0 0 0 0 Done 302 0 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5229 4927 5229 Downstairs last five completed: 4927 4926 4925 4924 4923 Upstairs last five completed: 4230 4229 4228 4227 4226 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.808826736Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.809001638Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.810934331Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.810970602Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.810994311Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 4927","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.811282202Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.811316247Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.811339701Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.811362155Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:22.811390826Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 5229 5228 5227 5226 5225 Upstairs last five completed: 4230 4229 4228 4227 4226 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:33:26.427 INFO request completed, latency_us: 462, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 682e7a30-5c7f-4c42-b26f-0dd8cac334e0, remote_addr: 127.0.0.1:37434, [8810] Got stop action so:false kr:false local_addr: 127.0.0.1:9998 Jan 02 22:33:26.428 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1a43615d-f311-4ebc-90f8-ecb473025abd, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:26.428865148Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:26.428981576Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:26.429028154Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:26.429051661Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:26.42916087Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:26.429298205Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:27.430266124Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:27.43033266Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:27.430360868Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:27.430387329Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:27.430407801Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:27.430440497Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:27.43050485Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:28.431359898Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:28.431424638Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:28.43145258Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:28.431479244Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:28.431499561Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:28.431529965Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:28.431621705Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:29.432404004Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:29.432475201Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:29.432503077Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:29.432529449Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:29.43256073Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:29.432604066Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:29.432665432Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:33:30.430 INFO request completed, latency_us: 241, response_code: 200, uri: /state/cid/0, method: GET, req_id: efc4f7de-386e-4e53-b567-7bfc3636de08, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 3787 len 32768 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:30.433238049Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:30.433302744Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:30.433331417Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:30.434113507Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:30.434141761Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 1 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:30.434165549Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:30.434497441Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 002/300 Write block 2262 len 8192 data: 5 3 003/300 Read block 1555 len 8192 004/300 Read block 762 len 8192 005/300 Write block 1703 len 20480 data: 2 3 4 4 3 006/300 Read block 2625 len 20480 007/300 Write block 1587 len 32768 data: 3 3 4 4 6 6 5 5 008/300 Read block 2163 len 40960 009/300 Read block 1497 len 36864 010/300 Read block 3159 len 36864 011/300 Read block 2019 len 16384 012/300 Flush 013/300 Read block 1690 len 40960 014/300 Read block 3190 len 40960 015/300 Read block 3819 len 16384 016/300 Write block 4963 len 12288 data: 3 3 4 017/300 Write block 4125 len 12288 data: 5 5 7 018/300 Read block 1989 len 32768 019/300 Read block 4948 len 8192 020/300 Read block 3319 len 28672 021/300 Read block 101 len 12288 022/300 Read block 4198 len 24576 023/300 Write block 1484 len 32768 data: 2 2 2 2 3 3 3 4 024/300 Write block 4589 len 12288 data: 4 4 4 025/300 Write block 4100 len 24576 data: 4 4 4 2 2 2 026/300 Write block 209 len 32768 data: 5 3 3 3 4 4 4 5 027/300 Write block 3043 len 32768 data: 3 3 3 4 5 7 4 5 028/300 Write block 1627 len 20480 data: 5 3 4 4 5 029/300 Read block 4630 len 40960 030/300 Read block 2104 len 32768 031/300 Write block 2822 len 40960 data: 7 7 6 7 7 6 6 4 4 2 032/300 Write block 719 len 8192 data: 2 2 033/300 Read block 468 len 40960 034/300 Write block 2502 len 40960 data: 3 4 4 5 6 6 7 7 8 9 035/300 Read block 1315 len 40960 036/300 Read block 1379 len 4096 037/300 Flush 038/300 Read block 116 len 40960 039/300 Read block 4702 len 24576 040/300 Read block 3872 len 16384 041/300 Read block 2490 len 16384 042/300 Read block 2744 len 32768 043/300 Write block 1533 len 12288 data: 3 3 3 044/300 Flush 045/300 Read block 3672 len 28672 046/300 Write block 1263 len 16384 data: 4 4 4 3 047/300 Read block 59 len 20480 048/300 Read block 1085 len 32768 049/300 Write block 3421 len 8192 data: 5 4 050/300 Read block 1377 len 36864 051/300 Write block 1506 len 24576 data: 5 5 6 5 6 5 052/300 Write block 3677 len 20480 data: 2 4 3 3 3 053/300 Read block 1501 len 16384 054/300 Read block 4518 len 28672 055/300 Read block 4429 len 12288 056/300 Write block 2409 len 36864 data: 4 4 5 6 7 7 6 6 6 057/300 Read block 683 len 16384 058/300 Read block 1666 len 4096 059/300 Read block 3229 len 24576 060/300 Read block 3031 len 8192 061/300 Read block 4582 len 36864 062/300 Flush 063/300 Flush 064/300 Read block 3502 len 36864 065/300 Read block 3037 len 4096 066/300 Write block 4218 len 32768 data: 6 5 5 4 4 4 4 4 067/300 Read block 764 len 24576 068/300 Flush 069/300 Read block 3919 len 4096 070/300 Read block 144 len 28672 071/300 Read block 4610 len 4096 072/300 Write block 324 len 24576 data: 5 5 4 4 4 4 073/300 Read block 46 len 28672 074/300 Write block 2387 len 20480 data: 4 4 4 4 5 075/300 Read block 36 len 20480 076/300 Read block 2859 len 32768 077/300 Read block 4492 len 16384 078/300 Read block 2855 len 12288 079/300 Write block 2850 len 8192 data: 4 5 080/300 Write block 2906 len 12288 data: 3 2 3 081/300 Read block 4358 len 40960 082/300 Write block 3198 len 40960 data: 3 3 3 3 2 2 2 2 2 2 083/300 Read block 236 len 8192 084/300 Write block 532 len 24576 data: 4 2 3 5 5 5 085/300 Read block 1931 len 12288 086/300 Read block 2369 len 28672 087/300 Read block 2574 len 28672 088/300 Read block 217 len 28672 089/300 Write block 155 len 4096 data: 4 090/300 Read block 4232 len 12288 091/300 Write block 3309 len 16384 data: 3 4 4 2 092/300 Read block 1236 len 12288 093/300 Write block 924 len 24576 data: 5 4 4 4 4 4 094/300 Read block 3855 len 16384 095/300 Flush 096/300 Write block 1635 len 40960 data: 4 4 3 3 5 4 4 3 3 4 097/300 Read block 942 len 24576 098/300 Read block 1719 len 32768 099/300 Read block 3711 len 24576 100/300 Read block 3980 len 32768 101/300 Read block 1 len 24576 102/300 Write block 713 len 12288 data: 4 4 3 103/300 Read block 3637 len 4096 104/300 Read block 47 len 12288 105/300 Read block 2285 len 16384 106/300 Write block 1114 len 36864 data: 6 6 4 3 3 3 3 4 3 107/300 Write block 4934 len 40960 data: 6 7 6 4 5 3 2 2 2 2 108/300 Write block 2415 len 40960 data: 7 7 7 7 6 6 5 3 3 2 109/300 Write block 105 len 32768 data: 6 6 4 3 3 3 2 2 110/300 Read block 4024 len 40960 111/300 Write block 3246 len 4096 data: 4 112/300 Read block 2971 len 16384 113/300 Write block 1007 len 32768 data: 3 3 3 3 4 4 4 4 114/300 Read block 4739 len 32768 115/300 Write block 2877 len 28672 data: 2 2 2 3 3 2 2 116/300 Read block 4539 len 8192 117/300 Read block 31 len 12288 118/300 Flush 119/300 Write block 864 len 4096 data: 4 120/300 Read block 1936 len 40960 121/300 Read block 1623 len 20480 122/300 Read block 1159 len 28672 123/300 Read block 3205 len 12288 124/300 Read block 1331 len 40960 125/300 Read block 3562 len 16384 126/300 Read block 4342 len 40960 127/300 Read block 2409 len 40960 128/300 Flush 129/300 Read block 3193 len 8192 130/300 Read block 3576 len 40960 131/300 Write block 1682 len 16384 data: 4 5 4 5 132/300 Write block 1953 len 28672 data: 2 2 2 2 2 2 2 133/300 Read block 2039 len 8192 134/300 Write block 534 len 12288 data: 4 6 6 135/300 Read block 2783 len 36864 136/300 Write block 469 len 40960 data: 3 4 3 4 4 5 5 5 5 5 137/300 Read block 1722 len 12288 138/300 Read block 1558 len 12288 139/300 Flush 140/300 Read block 90 len 32768 141/300 Read block 323 len 4096 142/300 Write block 4234 len 32768 data: 6 5 5 4 4 3 3 3 143/300 Read block 2683 len 4096 144/300 Read block 2298 len 20480 145/300 Flush 146/300 Read block 2990 len 32768 147/300 Read block 3658 len 36864 148/300 Flush 149/300 Read block 1969 len 28672 150/300 Write block 93 len 40960 data: 4 6 4 4 4 5 5 5 5 3 151/300 Flush 152/300 Read block 3277 len 36864 153/300 Read block 361 len 36864 154/300 Read block 3960 len 36864 155/300 Read block 3795 len 40960 156/300 Read block 3581 len 40960 157/300 Read block 1659 len 36864 158/300 Write block 2516 len 16384 data: 6 6 5 3 159/300 Flush 160/300 Flush 161/300 Write block 3012 len 20480 data: 3 5 6 6 5 162/300 Read block 4301 len 28672 163/300 Write block 2260 len 24576 data: 5 5 6 4 4 3 164/300 Read block 3698 len 32768 165/300 Write block 4786 len 8192 data: 2 3 166/300 Write block 1567 len 12288 data: 2 2 2 167/300 Write block 2624 len 28672 data: 5 5 5 7 7 6 4 168/300 Read block 1187 len 36864 169/300 Read block 3209 len 12288 170/300 Write block 2650 len 32768 data: 4 6 5 5 5 5 4 3 171/300 Flush 172/300 Read block 66 len 4096 173/300 Read block 4534 len 28672 174/300 Write block 8 len 40960 data: 3 4 6 8 6 6 7 7 7 6 175/300 Flush 176/300 Flush 177/300 Read block 4990 len 28672 178/300 Read block 1375 len 12288 179/300 Read block 589 len 40960 180/300 Read block 399 len 40960 181/300 Read block 2391 len 16384 182/300 Read block 2981 len 16384 183/300 Read block 3456 len 40960 184/300 Read block 3494 len 12288 185/300 Read block 1085 len 4096 186/300 Write block 2705 len 16384 data: 4 4 2 3 187/300 Read block 1819 len 24576 188/300 Write block 2586 len 24576 data: 3 3 3 3 3 2 189/300 Write block 2906 len 8192 data: 4 3 190/300 Read block 2755 len 32768 191/300 Flush 192/300 Write block 4348 len 20480 data: 3 2 2 2 2 193/300 Write block 2309 len 16384 data: 4 4 3 4 194/300 Write block 4782 len 28672 data: 2 2 2 2 3 4 3 195/300 Read block 615 len 36864 196/300 Read block 3766 len 20480 197/300 Read block 960 len 4096 198/300 Write block 2613 len 36864 data: 5 5 6 6 5 6 6 6 6 199/300 Write block 1567 len 36864 data: 3 3 3 2 2 2 2 2 2 200/300 Write block 4248 len 40960 data: 5 5 5 4 4 4 4 4 6 6 201/300 Write block 364 len 36864 data: 5 4 5 6 3 3 2 2 3 202/300 Read block 2998 len 24576 203/300 Write block 2446 len 24576 data: 5 5 4 3 3 3 204/300 Write block 133 len 8192 data: 4 5 205/300 Write block 1495 len 40960 data: 7 5 4 4 5 5 4 2 4 5 206/300 Read block 2677 len 28672 207/300 Write block 4245 len 20480 data: 4 4 4 6 6 208/300 Flush 209/300 Read block 4455 len 36864 210/300 Write block 1188 len 8192 data: 4 4 211/300 Flush 212/300 Read block 216 len 40960 213/300 Read block 3276 len 8192 214/300 Read block 1965 len 16384 215/300 Read block 2110 len 20480 216/300 Read block 3918 len 16384 217/300 Write block 3253 len 20480 data: 4 5 5 5 5 218/300 Flush 219/300 Read block 2200 len 4096 220/300 Write block 1581 len 24576 data: 3 3 3 3 6 4 221/300 Read block 4804 len 36864 222/300 Write block 1479 len 4096 data: 2 223/300 Write block 4202 len 32768 data: 4 4 5 5 4 4 4 5 224/300 Read block 4419 len 20480 225/300 Flush 226/300 Write block 421 len 16384 data: 6 5 4 3 227/300 Write block 1391 len 24576 data: 5 8 7 6 6 6 228/300 Write block 459 len 24576 data: 4 4 4 4 4 3 229/300 Read block 1437 len 12288 230/300 Read block 2727 len 20480 231/300 Read block 4058 len 40960 232/300 Read block 2230 len 32768 233/300 Read block 857 len 12288 234/300 Read block 2378 len 12288 235/300 Read block 351 len 8192 236/300 Write block 3647 len 16384 data: 2 2 3 3 237/300 Write block 1253 len 28672 data: 4 4 4 5 6 6 5 238/300 Read block 2426 len 40960 239/300 Read block 1236 len 40960 240/300 Write block 1168 len 16384 data: 3 4 3 2 241/300 Read block 4297 len 28672 242/300 Read block 2887 len 16384 243/300 Write block 1160 len 28672 data: 4 4 4 4 4 4 5 244/300 Write block 2431 len 20480 data: 3 2 3 4 4 245/300 Write block 1296 len 40960 data: 4 4 5 4 5 4 4 3 3 3 246/300 Write block 1348 len 28672 data: 6 6 7 5 5 6 5 247/300 Write block 3959 len 8192 data: 4 4 248/300 Read block 3498 len 12288 249/300 Read block 4811 len 20480 250/300 Write block 2267 len 32768 data: 4 5 5 4 4 4 4 4 251/300 Read block 1903 len 8192 252/300 Read block 4377 len 32768 253/300 Read block 2836 len 24576 254/300 Write block 2688 len 20480 data: 5 4 4 4 4 255/300 Read block 2956 len 16384 256/300 Read block 3306 len 32768 257/300 Read block 4596 len 12288 258/300 Read block 807 len 28672 259/300 Read block 1264 len 16384 260/300 Write block 660 len 32768 data: 6 5 5 8 8 8 6 6 261/300 Write block 4523 len 24576 data: 3 3 3 2 2 3 262/300 Write block 1995 len 8192 data: 2 2 263/300 Write block 564 len 8192 data: 3 3 264/300 Read block 694 len 8192 265/300 Write block 1255 len 36864 data: 5 6 7 7 6 6 6 4 5 266/300 Flush 267/300 Flush 268/300 Read block 658 len 40960 269/300 Read block 1252 len 20480 270/300 Read block 4447 len 16384 271/300 Read block 3013 len 8192 272/300 Write block 4502 len 20480 data: 4 5 5 4 4 273/300 Flush 274/300 Flush 275/300 Write block 851 len 16384 data: 6 5 5 4 276/300 Read block 1297 len 8192 277/300 Write block 164 len 4096 data: 3 278/300 Write block 1079 len 16384 data: 5 4 4 4 279/300 Read block 166 len 36864 280/300 Write block 2635 len 20480 data: 7 6 6 6 6 281/300 Write block 884 len 16384 data: 4 4 3 3 282/300 Read block 72 len 40960 283/300 Read block 2547 len 20480 284/300 Flush 285/300 Read block 4155 len 36864 286/300 Read block 1133 len 28672 287/300 Flush 288/300 Write block 1816 len 28672 data: 2 2 2 2 2 2 2 289/300 Write block 1438 len 4096 data: 3 290/300 Write block 2434 len 36864 data: 5 5 4 3 4 4 3 3 3 291/300 Read block 2922 len 12288 292/300 Write block 3001 len 20480 data: 2 2 2 2 2 293/300 Read block 2945 len 20480 294/300 Read block 2267 len 16384 295/300 Read block 257 len 12288 296/300 Read block 1639 len 16384 297/300 Read block 427 len 4096 298/300 Write block 2787 len 32768 data: 4 4 4 3 4 4 4 3 299/300 Write block 829 len 28672 data: 4 3 3 3 3 5 6 300/300 Read block 1337 len 24576 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:33:31.072 INFO request completed, latency_us: 291, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4bc04bf0-e104-4afa-bbe2-419f5ba6e038, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 4231 Acked 5230 Read 8 New Done Done true 4232 Acked 5231 Write 2 New Done Done false 4233 Acked 5232 Read 2 New Done Done false 4234 Acked 5233 Read 2 New Done Done false 4235 Acked 5234 Write 5 New Done Done false 4236 Acked 5235 Read 5 New Done Done false 4237 Acked 5236 Write 8 New Done Done false 4238 Acked 5237 Read 10 New Done Done false 4239 Acked 5238 Read 9 New Done Done false 4240 Acked 5239 Read 9 New Done Done false 4241 Acked 5240 Read 4 New Done Done false 4242 Acked 5241 Flush 0 New Done Done false 4243 Acked 5242 Read 10 New Done Done false 4244 Acked 5243 Read 10 New Done Done false 4245 Acked 5244 Read 4 New Done Done false 4246 Acked 5245 Write 3 New Done Done false 4247 Acked 5246 Write 3 New Done Done false 4248 Acked 5247 Read 8 New Done Done false 4249 Acked 5248 Read 2 New Done Done false 4250 Acked 5249 Read 7 New Done Done false 4251 Acked 5250 Read 3 New Done Done false 4252 Acked 5251 Read 6 New Done Done false 4253 Acked 5252 Write 8 New Done Done false 4254 Acked 5253 Write 3 New Done Done false 4255 Acked 5254 Write 6 New Done Done false 4256 Acked 5255 Write 8 New Done Done false 4257 Acked 5256 Write 8 New Done Done false 4258 Acked 5257 Write 5 New Done Done false 4259 Acked 5258 Read 10 New Done Done false 4260 Acked 5259 Read 8 New Done Done false 4261 Acked 5260 Write 10 New Done Done false 4262 Acked 5261 Write 2 New Done Done false 4263 Acked 5262 Read 10 New Done Done false 4264 Acked 5263 Write 10 New Done Done false 4265 Acked 5264 Read 10 New Done Done false 4266 Acked 5265 Read 1 New Done Done false 4267 Acked 5266 Flush 0 New Done Done false 4268 Acked 5267 Read 10 New Done Done false 4269 Acked 5268 Read 6 New Done Done false 4270 Acked 5269 Read 4 New Done Done false 4271 Acked 5270 Read 4 New Done Done false 4272 Acked 5271 Read 8 New Done Done false 4273 Acked 5272 Write 3 New Done Done false 4274 Acked 5273 Flush 0 New Done Done false 4275 Acked 5274 Read 7 New Done Done false 4276 Acked 5275 Write 4 New Done Done false 4277 Acked 5276 Read 5 New Done Done false 4278 Acked 5277 Read 8 New Done Done false 4279 Acked 5278 Write 2 New Done Done false 4280 Acked 5279 Read 9 New Done Done false 4281 Acked 5280 Write 6 New Done Done false 4282 Acked 5281 Write 5 New Done Done false 4283 Acked 5282 Read 4 New Done Done false 4284 Acked 5283 Read 7 New Done Done false 4285 Acked 5284 Read 3 New Done Done false 4286 Acked 5285 Write 9 New Done Done false 4287 Acked 5286 Read 4 New Done Done false 4288 Acked 5287 Read 1 New Done Done false 4289 Acked 5288 Read 6 New Done Done false 4290 Acked 5289 Read 2 New Done Done false 4291 Acked 5290 Read 9 New Done Done false 4292 Acked 5291 Flush 0 New Done Done false 4293 Acked 5292 Flush 0 New Done Done false 4294 Acked 5293 Read 9 New Done Done false 4295 Acked 5294 Read 1 New Done Done false 4296 Acked 5295 Write 8 New Done Done false 4297 Acked 5296 Read 6 New Done Done false 4298 Acked 5297 Flush 0 New Done Done false 4299 Acked 5298 Read 1 New Done Done false 4300 Acked 5299 Read 7 New Done Done false 4301 Acked 5300 Read 1 New Done Done false 4302 Acked 5301 Write 6 New Done Done false 4303 Acked 5302 Read 7 New Done Done false 4304 Acked 5303 Write 5 New Done Done false 4305 Acked 5304 Read 5 New Done Done false 4306 Acked 5305 Read 8 New Done Done false 4307 Acked 5306 Read 4 New Done Done false 4308 Acked 5307 Read 3 New Done Done false 4309 Acked 5308 Write 2 New Done Done false 4310 Acked 5309 Write 3 New Done Done false 4311 Acked 5310 Read 10 New Done Done false 4312 Acked 5311 Write 10 New Done Done false 4313 Acked 5312 Read 2 New Done Done false 4314 Acked 5313 Write 6 New Done Done false 4315 Acked 5314 Read 3 New Done Done false 4316 Acked 5315 Read 7 New Done Done false 4317 Acked 5316 Read 7 New Done Done false 4318 Acked 5317 Read 7 New Done Done false 4319 Acked 5318 Write 1 New Done Done false 4320 Acked 5319 Read 3 New Done Done false 4321 Acked 5320 Write 4 New Done Done false 4322 Acked 5321 Read 3 New Done Done false 4323 Acked 5322 Write 6 New Done Done false 4324 Acked 5323 Read 4 New Done Done false 4325 Acked 5324 Flush 0 New Done Done false 4326 Acked 5325 Write 10 New Done Done false 4327 Acked 5326 Read 6 New Done Done false 4328 Acked 5327 Read 8 New Done Done false 4329 Acked 5328 Read 6 New Done Done false 4330 Acked 5329 Read 8 New Done Done false 4331 Acked 5330 Read 6 New Done Done false 4332 Acked 5331 Flush 0 New Done Done false 4333 Acked 5332 Write 3 New Done Done false 4334 Acked 5333 Read 1 New Done Done false 4335 Acked 5334 Read 3 New Done Done false 4336 Acked 5335 Read 4 New Done Done false 4337 Acked 5336 Write 9 New Done Done false 4338 Acked 5337 Write 10 New Done Done false 4339 Acked 5338 Write 10 New Done Done false 4340 Acked 5339 Write 8 New Done Done false 4341 Acked 5340 Read 10 New Done Done false 4342 Acked 5341 Write 1 New Done Done false 4343 Acked 5342 Read 4 New Done Done false 4344 Acked 5343 Write 8 New Done Done false 4345 Acked 5344 Read 8 New Done Done false 4346 Acked 5345 Write 7 New Done Done false 4347 Acked 5346 Read 2 New Done Done false 4348 Acked 5347 Read 3 New Done Done false 4349 Acked 5348 Flush 0 New Done Done false 4350 Acked 5349 Write 1 New Done Done false 4351 Acked 5350 Read 10 New Done Done false 4352 Acked 5351 Read 5 New Done Done false 4353 Acked 5352 Read 7 New Done Done false 4354 Acked 5353 Read 3 New Done Done false 4355 Acked 5354 Read 10 New Done Done false 4356 Acked 5355 Read 4 New Done Done false 4357 Acked 5356 Read 10 New Done Done false 4358 Acked 5357 Read 10 New Done Done false 4359 Acked 5358 Flush 0 New Done Done false 4360 Acked 5359 Read 2 New Done Done false 4361 Acked 5360 Read 10 New Done Done false 4362 Acked 5361 Write 4 New Done Done false 4363 Acked 5362 Write 7 New Done Done false 4364 Acked 5363 Read 2 New Done Done false 4365 Acked 5364 Write 3 New Done Done false 4366 Acked 5365 Read 9 New Done Done false 4367 Acked 5366 Write 10 New Done Done false 4368 Acked 5367 Read 3 New Done Done false 4369 Acked 5368 Read 3 New Done Done false 4370 Acked 5369 Flush 0 New Done Done false 4371 Acked 5370 Read 8 New Done Done false 4372 Acked 5371 Read 1 New Done Done false 4373 Acked 5372 Write 8 New Done Done false 4374 Acked 5373 Read 1 New Done Done false 4375 Acked 5374 Read 5 New Done Done false 4376 Acked 5375 Flush 0 New Done Done false 4377 Acked 5376 Read 8 New Done Done false 4378 Acked 5377 Read 9 New Done Done false 4379 Acked 5378 Flush 0 New Done Done false 4380 Acked 5379 Read 7 New Done Done false 4381 Acked 5380 Write 10 New Done Done false 4382 Acked 5381 Flush 0 New Done Done false 4383 Acked 5382 Read 9 New Done Done false 4384 Acked 5383 Read 9 New Done Done false 4385 Acked 5384 Read 9 New Done Done false 4386 Acked 5385 Read 10 New Done Done false 4387 Acked 5386 Read 10 New Done Done false 4388 Acked 5387 Read 9 New Done Done false 4389 Acked 5388 Write 4 New Done Done false 4390 Acked 5389 Flush 0 New Done Done false 4391 Acked 5390 Flush 0 New Done Done false 4392 Acked 5391 Write 5 New Done Done false 4393 Acked 5392 Read 7 New Done Done false 4394 Acked 5393 Write 6 New Done Done false 4395 Acked 5394 Read 8 New Done Done false 4396 Acked 5395 Write 2 New Done Done false 4397 Acked 5396 Write 3 New Done Done false 4398 Acked 5397 Write 7 New Done Done false 4399 Acked 5398 Read 9 New Done Done false 4400 Acked 5399 Read 3 New Done Done false 4401 Acked 5400 Write 8 New Done Done false 4402 Acked 5401 Flush 0 New Done Done false 4403 Acked 5402 Read 1 New Done Done false 4404 Acked 5403 Read 7 New Done Done false 4405 Acked 5404 Write 10 New Done Done false 4406 Acked 5405 Flush 0 New Done Done false 4407 Acked 5406 Flush 0 New Done Done false 4408 Acked 5407 Read 7 New Done Done false 4409 Acked 5408 Read 3 New Done Done false 4410 Acked 5409 Read 10 New Done Done false 4411 Acked 5410 Read 10 New Done Done false 4412 Acked 5411 Read 4 New Done Done false 4413 Acked 5412 Read 4 New Done Done false 4414 Acked 5413 Read 10 New Done Done false 4415 Acked 5414 Read 3 New Done Done false 4416 Acked 5415 Read 1 New Done Done false 4417 Acked 5416 Write 4 New Done Done false 4418 Acked 5417 Read 6 New Done Done false 4419 Acked 5418 Write 6 New Done Done false 4420 Acked 5419 Write 2 New Done Done false 4421 Acked 5420 Read 8 New Done Done false 4422 Acked 5421 Flush 0 New Done Done false 4423 Acked 5422 Write 5 New Done Done false 4424 Acked 5423 Write 4 New Done Done false 4425 Acked 5424 Write 7 New Done Done false 4426 Acked 5425 Read 9 New Done Done false 4427 Acked 5426 Read 5 New Done Done false 4428 Acked 5427 Read 1 New Done Done false 4429 Acked 5428 Write 9 New Done Done false 4430 Acked 5429 Write 9 New Done Done false 4431 Acked 5430 Write 10 New Done Done false 4432 Acked 5431 Write 9 New Done Done false 4433 Acked 5432 Read 6 New Done Done false 4434 Acked 5433 Write 6 New Done Done false 4435 Acked 5434 Write 2 New Done Done false 4436 Acked 5435 Write 10 New Done Done false 4437 Acked 5436 Read 7 New Done Done false 4438 Acked 5437 Write 5 New Done Done false 4439 Acked 5438 Flush 0 New Done Done false 4440 Acked 5439 Read 9 New Done Done false 4441 Acked 5440 Write 2 New Done Done false 4442 Acked 5441 Flush 0 New Done Done false 4443 Acked 5442 Read 10 New Done Done false 4444 Acked 5443 Read 2 New Done Done false 4445 Acked 5444 Read 4 New Done Done false 4446 Acked 5445 Read 5 New Done Done false 4447 Acked 5446 Read 4 New Done Done false 4448 Acked 5447 Write 5 New Done Done false 4449 Acked 5448 Flush 0 New Done Done false 4450 Acked 5449 Read 1 New Done Done false 4451 Acked 5450 Write 6 New Done Done false 4452 Acked 5451 Read 9 New Done Done false 4453 Acked 5452 Write 1 New Done Done false 4454 Acked 5453 Write 8 New Done Done false 4455 Acked 5454 Read 5 New Done Done false 4456 Acked 5455 Flush 0 New Done Done false 4457 Acked 5456 Write 4 New Done Done false 4458 Acked 5457 Write 6 New Done Done false 4459 Acked 5458 Write 6 New Done Done false 4460 Acked 5459 Read 3 New Done Done false 4461 Acked 5460 Read 5 New Done Done false 4462 Acked 5461 Read 10 New Done Done false 4463 Acked 5462 Read 8 New Done Done false 4464 Acked 5463 Read 3 New Done Done false 4465 Acked 5464 Read 3 New Done Done false 4466 Acked 5465 Read 2 New Done Done false 4467 Acked 5466 Write 4 New Done Done false 4468 Acked 5467 Write 7 New Done Done false 4469 Acked 5468 Read 10 New Done Done false 4470 Acked 5469 Read 10 New Done Done false 4471 Acked 5470 Write 4 New Done Done false 4472 Acked 5471 Read 7 New Done Done false 4473 Acked 5472 Read 4 New Done Done false 4474 Acked 5473 Write 7 New Done Done false 4475 Acked 5474 Write 5 New Done Done false 4476 Acked 5475 Write 10 New Done Done false 4477 Acked 5476 Write 7 New Done Done false 4478 Acked 5477 Write 2 New Done Done false 4479 Acked 5478 Read 3 New Done Done false 4480 Acked 5479 Read 5 New Done Done false 4481 Acked 5480 Write 8 New Done Done false 4482 Acked 5481 Read 2 New Done Done false 4483 Acked 5482 Read 8 New Done Done false 4484 Acked 5483 Read 6 New Done Done false 4485 Acked 5484 Write 5 New Done Done false 4486 Acked 5485 Read 4 New Done Done false 4487 Acked 5486 Read 8 New Done Done false 4488 Acked 5487 Read 3 New Done Done false 4489 Acked 5488 Read 7 New Done Done false 4490 Acked 5489 Read 4 New Done Done false 4491 Acked 5490 Write 8 New Done Done false 4492 Acked 5491 Write 6 New Done Done false 4493 Acked 5492 Write 2 New Done Done false 4494 Acked 5493 Write 2 New Done Done false 4495 Acked 5494 Read 2 New Done Done false 4496 Acked 5495 Write 9 New Done Done false 4497 Acked 5496 Flush 0 New Done Done false 4498 Acked 5497 Flush 0 New Done Done false 4499 Acked 5498 Read 10 New Done Done false 4500 Acked 5499 Read 5 New Done Done false 4501 Acked 5500 Read 4 New Done Done false 4502 Acked 5501 Read 2 New Done Done false 4503 Acked 5502 Write 5 New Done Done false 4504 Acked 5503 Flush 0 New Done Done false 4505 Acked 5504 Flush 0 New Done Done false 4506 Acked 5505 Write 4 New Done Done false 4507 Acked 5506 Read 2 New Done Done false 4508 Acked 5507 Write 1 New Done Done false 4509 Acked 5508 Write 4 New Done Done false 4510 Acked 5509 Read 9 New Done Done false 4511 Acked 5510 Write 5 New Done Done false 4512 Acked 5511 Write 4 New Done Done false 4513 Acked 5512 Read 10 New Done Done false 4514 Acked 5513 Read 5 New Done Done false 4515 Acked 5514 Flush 0 New Done Done false 4516 Acked 5515 Read 9 New Done Done false 4517 Acked 5516 Read 7 New Done Done false 4518 Acked 5517 Flush 0 New Done Done false 4519 Acked 5518 Write 7 New Done Done false 4520 Acked 5519 Write 1 New Done Done false 4521 Acked 5520 Write 9 New Done Done false 4522 Acked 5521 Read 3 New Done Done false 4523 Acked 5522 Write 5 New Done Done false 4524 Acked 5523 Read 5 New Done Done false 4525 Acked 5524 Read 4 New Done Done false 4526 Acked 5525 Read 3 New Done Done false 4527 Acked 5526 Read 4 New Done Done false 4528 Acked 5527 Read 1 New Done Done false 4529 Acked 5528 Write 8 New Done Done false 4530 Acked 5529 Write 7 New Done Done false 4531 Acked 5530 Read 6 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5229 5517 5517 Downstairs last five completed: 5229 5228 5227 5226 5225 Upstairs last five completed: 4531 4530 4529 4528 4527 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:31.435903072Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:31.435971351Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:31.436009523Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:31.436034764Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:31.436061039Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:31.436081327Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:31.436108547Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:31.436304433Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:32.43803406Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:32.438099962Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:32.438128661Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:32.438155466Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:32.438175857Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:32.438210736Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:32.438436493Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:33.439254098Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:33.439336844Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:33.439366092Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:33.439392703Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:33.439413162Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:33.439437017Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:33.439693411Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:34.440562323Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:34.440629183Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:34.440657166Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:34.440683816Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:34.440704285Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:34.440729192Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:34.441007698Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 4231 Acked 5230 Read 8 New Done Done true 4232 Acked 5231 Write 2 New Done Done true 4233 Acked 5232 Read 2 New Done Done true 4234 Acked 5233 Read 2 New Done Done true 4235 Acked 5234 Write 5 New Done Done true 4236 Acked 5235 Read 5 New Done Done true 4237 Acked 5236 Write 8 New Done Done true 4238 Acked 5237 Read 10 New Done Done true 4239 Acked 5238 Read 9 New Done Done true 4240 Acked 5239 Read 9 New Done Done true 4241 Acked 5240 Read 4 New Done Done true 4242 Acked 5241 Flush 0 New Done Done true 4243 Acked 5242 Read 10 New Done Done true 4244 Acked 5243 Read 10 New Done Done true 4245 Acked 5244 Read 4 New Done Done true 4246 Acked 5245 Write 3 New Done Done true 4247 Acked 5246 Write 3 New Done Done true 4248 Acked 5247 Read 8 New Done Done true 4249 Acked 5248 Read 2 New Done Done true 4250 Acked 5249 Read 7 New Done Done true 4251 Acked 5250 Read 3 New Done Done true 4252 Acked 5251 Read 6 New Done Done true 4253 Acked 5252 Write 8 New Done Done true 4254 Acked 5253 Write 3 New Done Done true 4255 Acked 5254 Write 6 New Done Done true 4256 Acked 5255 Write 8 New Done Done true 4257 Acked 5256 Write 8 New Done Done true 4258 Acked 5257 Write 5 New Done Done true 4259 Acked 5258 Read 10 New Done Done true 4260 Acked 5259 Read 8 New Done Done true 4261 Acked 5260 Write 10 New Done Done true 4262 Acked 5261 Write 2 New Done Done true 4263 Acked 5262 Read 10 New Done Done true 4264 Acked 5263 Write 10 New Done Done true 4265 Acked 5264 Read 10 New Done Done true 4266 Acked 5265 Read 1 New Done Done true 4267 Acked 5266 Flush 0 New Done Done true 4268 Acked 5267 Read 10 New Done Done true 4269 Acked 5268 Read 6 New Done Done true 4270 Acked 5269 Read 4 New Done Done true 4271 Acked 5270 Read 4 New Done Done true 4272 Acked 5271 Read 8 New Done Done true 4273 Acked 5272 Write 3 New Done Done true 4274 Acked 5273 Flush 0 New Done Done true 4275 Acked 5274 Read 7 New Done Done true 4276 Acked 5275 Write 4 New Done Done true 4277 Acked 5276 Read 5 New Done Done true 4278 Acked 5277 Read 8 New Done Done true 4279 Acked 5278 Write 2 New Done Done true 4280 Acked 5279 Read 9 New Done Done true 4281 Acked 5280 Write 6 New Done Done true 4282 Acked 5281 Write 5 New Done Done true 4283 Acked 5282 Read 4 New Done Done true 4284 Acked 5283 Read 7 New Done Done true 4285 Acked 5284 Read 3 New Done Done true 4286 Acked 5285 Write 9 New Done Done true 4287 Acked 5286 Read 4 New Done Done true 4288 Acked 5287 Read 1 New Done Done true 4289 Acked 5288 Read 6 New Done Done true 4290 Acked 5289 Read 2 New Done Done true 4291 Acked 5290 Read 9 New Done Done true 4292 Acked 5291 Flush 0 New Done Done true 4293 Acked 5292 Flush 0 New Done Done true 4294 Acked 5293 Read 9 New Done Done true 4295 Acked 5294 Read 1 New Done Done true 4296 Acked 5295 Write 8 New Done Done true 4297 Acked 5296 Read 6 New Done Done true 4298 Acked 5297 Flush 0 New Done Done true 4299 Acked 5298 Read 1 New Done Done true 4300 Acked 5299 Read 7 New Done Done true 4301 Acked 5300 Read 1 New Done Done true 4302 Acked 5301 Write 6 New Done Done true 4303 Acked 5302 Read 7 New Done Done true 4304 Acked 5303 Write 5 New Done Done true 4305 Acked 5304 Read 5 New Done Done true 4306 Acked 5305 Read 8 New Done Done true 4307 Acked 5306 Read 4 New Done Done true 4308 Acked 5307 Read 3 New Done Done true 4309 Acked 5308 Write 2 New Done Done true 4310 Acked 5309 Write 3 New Done Done true 4311 Acked 5310 Read 10 New Done Done true 4312 Acked 5311 Write 10 New Done Done true 4313 Acked 5312 Read 2 New Done Done true 4314 Acked 5313 Write 6 New Done Done true 4315 Acked 5314 Read 3 New Done Done true 4316 Acked 5315 Read 7 New Done Done true 4317 Acked 5316 Read 7 New Done Done true 4318 Acked 5317 Read 7 New Done Done true 4319 Acked 5318 Write 1 New Done Done true 4320 Acked 5319 Read 3 New Done Done true 4321 Acked 5320 Write 4 New Done Done true 4322 Acked 5321 Read 3 New Done Done true 4323 Acked 5322 Write 6 New Done Done true 4324 Acked 5323 Read 4 New Done Done true 4325 Acked 5324 Flush 0 New Done Done true 4326 Acked 5325 Write 10 New Done Done true 4327 Acked 5326 Read 6 New Done Done true 4328 Acked 5327 Read 8 New Done Done true 4329 Acked 5328 Read 6 New Done Done true 4330 Acked 5329 Read 8 New Done Done true 4331 Acked 5330 Read 6 New Done Done true 4332 Acked 5331 Flush 0 New Done Done true 4333 Acked 5332 Write 3 New Done Done true 4334 Acked 5333 Read 1 New Done Done true 4335 Acked 5334 Read 3 New Done Done true 4336 Acked 5335 Read 4 New Done Done true 4337 Acked 5336 Write 9 New Done Done true 4338 Acked 5337 Write 10 New Done Done true 4339 Acked 5338 Write 10 New Done Done true 4340 Acked 5339 Write 8 New Done Done true 4341 Acked 5340 Read 10 New Done Done true 4342 Acked 5341 Write 1 New Done Done true 4343 Acked 5342 Read 4 New Done Done true 4344 Acked 5343 Write 8 New Done Done true 4345 Acked 5344 Read 8 New Done Done true 4346 Acked 5345 Write 7 New Done Done true 4347 Acked 5346 Read 2 New Done Done true 4348 Acked 5347 Read 3 New Done Done true 4349 Acked 5348 Flush 0 New Done Done true 4350 Acked 5349 Write 1 New Done Done true 4351 Acked 5350 Read 10 New Done Done true 4352 Acked 5351 Read 5 New Done Done true 4353 Acked 5352 Read 7 New Done Done true 4354 Acked 5353 Read 3 New Done Done true 4355 Acked 5354 Read 10 New Done Done true 4356 Acked 5355 Read 4 New Done Done true 4357 Acked 5356 Read 10 New Done Done true 4358 Acked 5357 Read 10 New Done Done true 4359 Acked 5358 Flush 0 New Done Done true 4360 Acked 5359 Read 2 New Done Done true 4361 Acked 5360 Read 10 New Done Done true 4362 Acked 5361 Write 4 New Done Done true 4363 Acked 5362 Write 7 New Done Done true 4364 Acked 5363 Read 2 New Done Done true 4365 Acked 5364 Write 3 New Done Done true 4366 Acked 5365 Read 9 New Done Done true 4367 Acked 5366 Write 10 New Done Done true 4368 Acked 5367 Read 3 New Done Done true 4369 Acked 5368 Read 3 New Done Done true 4370 Acked 5369 Flush 0 New Done Done true 4371 Acked 5370 Read 8 New Done Done true 4372 Acked 5371 Read 1 New Done Done true 4373 Acked 5372 Write 8 New Done Done true 4374 Acked 5373 Read 1 New Done Done true 4375 Acked 5374 Read 5 New Done Done true 4376 Acked 5375 Flush 0 New Done Done true 4377 Acked 5376 Read 8 New Done Done true 4378 Acked 5377 Read 9 New Done Done true 4379 Acked 5378 Flush 0 New Done Done true 4380 Acked 5379 Read 7 New Done Done true 4381 Acked 5380 Write 10 New Done Done true 4382 Acked 5381 Flush 0 New Done Done true 4383 Acked 5382 Read 9 New Done Done true 4384 Acked 5383 Read 9 New Done Done true 4385 Acked 5384 Read 9 New Done Done true 4386 Acked 5385 Read 10 New Done Done true 4387 Acked 5386 Read 10 New Done Done true 4388 Acked 5387 Read 9 New Done Done true 4389 Acked 5388 Write 4 New Done Done true 4390 Acked 5389 Flush 0 New Done Done true 4391 Acked 5390 Flush 0 New Done Done true 4392 Acked 5391 Write 5 New Done Done true 4393 Acked 5392 Read 7 New Done Done true 4394 Acked 5393 Write 6 New Done Done true 4395 Acked 5394 Read 8 New Done Done true 4396 Acked 5395 Write 2 New Done Done true 4397 Acked 5396 Write 3 New Done Done true 4398 Acked 5397 Write 7 New Done Done true 4399 Acked 5398 Read 9 New Done Done true 4400 Acked 5399 Read 3 New Done Done true 4401 Acked 5400 Write 8 New Done Done true 4402 Acked 5401 Flush 0 New Done Done true 4403 Acked 5402 Read 1 New Done Done true 4404 Acked 5403 Read 7 New Done Done true 4405 Acked 5404 Write 10 New Done Done true 4406 Acked 5405 Flush 0 New Done Done true 4407 Acked 5406 Flush 0 New Done Done true 4408 Acked 5407 Read 7 New Done Done true 4409 Acked 5408 Read 3 New Done Done true 4410 Acked 5409 Read 10 New Done Done true 4411 Acked 5410 Read 10 New Done Done true 4412 Acked 5411 Read 4 New Done Done true 4413 Acked 5412 Read 4 New Done Done true 4414 Acked 5413 Read 10 New Done Done true 4415 Acked 5414 Read 3 New Done Done true 4416 Acked 5415 Read 1 New Done Done true 4417 Acked 5416 Write 4 New Done Done true 4418 Acked 5417 Read 6 New Done Done true 4419 Acked 5418 Write 6 New Done Done true 4420 Acked 5419 Write 2 New Done Done true 4421 Acked 5420 Read 8 New Done Done true 4422 Acked 5421 Flush 0 New Done Done true 4423 Acked 5422 Write 5 New Done Done true 4424 Acked 5423 Write 4 New Done Done true 4425 Acked 5424 Write 7 New Done Done true 4426 Acked 5425 Read 9 New Done Done true 4427 Acked 5426 Read 5 New Done Done true 4428 Acked 5427 Read 1 New Done Done true 4429 Acked 5428 Write 9 New Done Done true 4430 Acked 5429 Write 9 New Done Done true 4431 Acked 5430 Write 10 New Done Done true 4432 Acked 5431 Write 9 New Done Done true 4433 Acked 5432 Read 6 New Done Done true 4434 Acked 5433 Write 6 New Done Done true 4435 Acked 5434 Write 2 New Done Done true 4436 Acked 5435 Write 10 New Done Done true 4437 Acked 5436 Read 7 New Done Done true 4438 Acked 5437 Write 5 New Done Done true 4439 Acked 5438 Flush 0 New Done Done true 4440 Acked 5439 Read 9 New Done Done true 4441 Acked 5440 Write 2 New Done Done true 4442 Acked 5441 Flush 0 New Done Done true 4443 Acked 5442 Read 10 New Done Done true 4444 Acked 5443 Read 2 New Done Done true 4445 Acked 5444 Read 4 New Done Done true 4446 Acked 5445 Read 5 New Done Done true 4447 Acked 5446 Read 4 New Done Done true 4448 Acked 5447 Write 5 New Done Done true 4449 Acked 5448 Flush 0 New Done Done true 4450 Acked 5449 Read 1 New Done Done true 4451 Acked 5450 Write 6 New Done Done true 4452 Acked 5451 Read 9 New Done Done true 4453 Acked 5452 Write 1 New Done Done true 4454 Acked 5453 Write 8 New Done Done true 4455 Acked 5454 Read 5 New Done Done true 4456 Acked 5455 Flush 0 New Done Done true 4457 Acked 5456 Write 4 New Done Done true 4458 Acked 5457 Write 6 New Done Done true 4459 Acked 5458 Write 6 New Done Done true 4460 Acked 5459 Read 3 New Done Done true 4461 Acked 5460 Read 5 New Done Done true 4462 Acked 5461 Read 10 New Done Done true 4463 Acked 5462 Read 8 New Done Done true 4464 Acked 5463 Read 3 New Done Done true 4465 Acked 5464 Read 3 New Done Done true 4466 Acked 5465 Read 2 New Done Done true 4467 Acked 5466 Write 4 New Done Done true 4468 Acked 5467 Write 7 New Done Done true 4469 Acked 5468 Read 10 New Done Done true 4470 Acked 5469 Read 10 New Done Done true 4471 Acked 5470 Write 4 New Done Done true 4472 Acked 5471 Read 7 New Done Done true 4473 Acked 5472 Read 4 New Done Done true 4474 Acked 5473 Write 7 New Done Done true 4475 Acked 5474 Write 5 New Done Done true 4476 Acked 5475 Write 10 New Done Done true 4477 Acked 5476 Write 7 New Done Done true 4478 Acked 5477 Write 2 New Done Done true 4479 Acked 5478 Read 3 New Done Done true 4480 Acked 5479 Read 5 New Done Done true 4481 Acked 5480 Write 8 New Done Done true 4482 Acked 5481 Read 2 New Done Done true 4483 Acked 5482 Read 8 New Done Done true 4484 Acked 5483 Read 6 New Done Done true 4485 Acked 5484 Write 5 New Done Done true 4486 Acked 5485 Read 4 New Done Done true 4487 Acked 5486 Read 8 New Done Done true 4488 Acked 5487 Read 3 New Done Done true 4489 Acked 5488 Read 7 New Done Done true 4490 Acked 5489 Read 4 New Done Done true 4491 Acked 5490 Write 8 New Done Done true 4492 Acked 5491 Write 6 New Done Done true 4493 Acked 5492 Write 2 New Done Done true 4494 Acked 5493 Write 2 New Done Done true 4495 Acked 5494 Read 2 New Done Done true 4496 Acked 5495 Write 9 New Done Done true 4497 Acked 5496 Flush 0 New Done Done true 4498 Acked 5497 Flush 0 New Done Done true 4499 Acked 5498 Read 10 New Done Done true 4500 Acked 5499 Read 5 New Done Done true 4501 Acked 5500 Read 4 New Done Done true 4502 Acked 5501 Read 2 New Done Done true 4503 Acked 5502 Write 5 New Done Done true 4504 Acked 5503 Flush 0 New Done Done true 4505 Acked 5504 Flush 0 New Done Done true 4506 Acked 5505 Write 4 New Done Done true 4507 Acked 5506 Read 2 New Done Done true 4508 Acked 5507 Write 1 New Done Done true 4509 Acked 5508 Write 4 New Done Done true 4510 Acked 5509 Read 9 New Done Done true 4511 Acked 5510 Write 5 New Done Done true 4512 Acked 5511 Write 4 New Done Done true 4513 Acked 5512 Read 10 New Done Done true 4514 Acked 5513 Read 5 New Done Done true 4515 Acked 5514 Flush 0 New Done Done true 4516 Acked 5515 Read 9 New Done Done true 4517 Acked 5516 Read 7 New Done Done true 4518 Acked 5517 Flush 0 New Done Done true 4519 Acked 5518 Write 7 New Done Done true 4520 Acked 5519 Write 1 New Done Done true 4521 Acked 5520 Write 9 New Done Done true 4522 Acked 5521 Read 3 New Done Done true 4523 Acked 5522 Write 5 New Done Done true 4524 Acked 5523 Read 5 New Done Done true 4525 Acked 5524 Read 4 New Done Done true 4526 Acked 5525 Read 3 New Done Done true 4527 Acked 5526 Read 4 New Done Done true 4528 Acked 5527 Read 1 New Done Done true 4529 Acked 5528 Write 8 New Done Done true 4530 Acked 5529 Write 7 New Done Done true 4531 Acked 5530 Read 6 New Done Done true 4532 Acked 5531 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5229 5531 5531 Downstairs last five completed: 5229 5228 5227 5226 5225 Upstairs last five completed: 4532 4531 4530 4529 4528 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:35.441216724Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:35.441331482Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:35.44139318Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:35.441440962Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:35.441471706Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:35.441495462Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:35.441752935Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:36.442490665Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:36.442555554Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:36.442583309Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:36.442607227Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:36.44263333Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:36.442653954Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:36.442676476Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:36.442908008Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1200) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.443712492Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.443889373Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.445776396Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.445819627Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.44584436Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 5229","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.446107486Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.446138164Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.44616149Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.446184053Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:37.446206326Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 5531 5530 5529 5528 5527 Upstairs last five completed: 4532 4531 4530 4529 4528 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:33:39.080 INFO request completed, latency_us: 462, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 35415b18-9d46-479c-ad00-5b3d0dd01334[8810] Got stop action so:false kr:false , remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:33:39.081 INFO request completed, latency_us: 318, response_code: 200, uri: /state/cid/0, method: GET, req_id: 51f6053f-35b5-4b98-b072-ea467164ce34, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:39.082061703Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:39.08216673Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:39.082222487Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:39.082278581Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:39.082327441Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:39.082374688Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:40.083488139Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:40.083552964Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:40.08358122Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:40.083608455Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:40.083629211Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:40.083683705Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:40.083787146Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:41.08483569Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:41.084911623Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:41.084941557Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:41.084968522Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:41.084998652Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:41.085030069Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:41.085070173Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:42.086801918Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:42.086868075Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:42.086895992Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:42.086920157Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:42.0869466Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:42.086967601Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:42.087014403Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:42.087118551Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:33:43.084 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/0, method: GET, req_id: a12d19e3-c661-4aa8-b690-eb07f11f67d0, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 1224 len 40960 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:43.088861652Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:43.088937973Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40002/300 Read block 3218 len 16384 ,"time":"2024-01-02T22:33:43.088968324Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:43.088999161Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:43.089019692Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 1 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:43.089042346Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:43.08927663Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 3838 len 8192 004/300 Flush 005/300 Read block 1647 len 40960 006/300 Flush 007/300 Read block 3886 len 12288 008/300 Read block 3783 len 32768 009/300 Write block 1759 len 36864 data: 6 5 3 5 4 4 3 4 4 010/300 Write block 571 len 28672 data: 3 3 3 2 2 3 3 011/300 Write block 2137 len 20480 data: 3 4 5 5 6 012/300 Read block 4763 len 8192 013/300 Read block 4220 len 16384 014/300 Write block 4058 len 40960 data: 2 2 2 2 2 2 2 2 2 2 015/300 Read block 2335 len 40960 016/300 Write block 4583 len 4096 data: 6 017/300 Write block 656 len 28672 data: 5 5 6 6 7 6 6 018/300 Write block 2630 len 4096 data: 5 019/300 Read block 2063 len 24576 020/300 Read block 4789 len 4096 021/300 Flush 022/300 Write block 4295 len 4096 data: 5 023/300 Write block 4164 len 12288 data: 4 4 4 024/300 Read block 1875 len 24576 025/300 Write block 362 len 28672 data: 4 5 6 5 6 7 4 026/300 Write block 3974 len 20480 data: 2 4 4 5 5 027/300 Read block 3088 len 4096 028/300 Read block 597 len 40960 029/300 Write block 4547 len 16384 data: 4 4 3 3 030/300 Write block 2055 len 12288 data: 4 5 5 031/300 Flush 032/300 Read block 2915 len 24576 033/300 Read block 3242 len 16384 034/300 Write block 2152 len 28672 data: 3 3 4 5 5 5 7 035/300 Read block 1666 len 16384 036/300 Read block 2785 len 20480 037/300 Read block 3106 len 12288 038/300 Write block 555 len 16384 data: 2 2 2 2 039/300 Flush 040/300 Write block 1129 len 4096 data: 5 041/300 Read block 1479 len 16384 042/300 Write block 593 len 40960 data: 2 4 4 3 4 4 4 4 4 4 043/300 Read block 2533 len 8192 044/300 Write block 3611 len 24576 data: 6 3 3 3 3 4 045/300 Write block 845 len 40960 data: 5 6 6 6 6 7 7 6 6 5 046/300 Read block 1978 len 8192 047/300 Flush 048/300 Write block 2609 len 8192 data: 5 5 049/300 Read block 2582 len 24576 050/300 Write block 2893 len 24576 data: 4 4 4 5 5 6 051/300 Read block 2207 len 28672 052/300 Read block 1083 len 36864 053/300 Read block 3405 len 4096 054/300 Write block 4302 len 36864 data: 2 2 3 2 2 4 3 3 3 055/300 Read block 1520 len 24576 056/300 Write block 1759 len 16384 data: 7 6 4 6 057/300 Flush 058/300 Read block 1375 len 32768 059/300 Read block 2669 len 4096 060/300 Read block 545 len 28672 061/300 Flush 062/300 Read block 1556 len 16384 063/300 Read block 553 len 20480 064/300 Write block 717 len 24576 data: 3 2 3 3 2 2 065/300 Flush 066/300 Write block 1232 len 36864 data: 4 3 2 2 2 2 2 2 3 067/300 Flush 068/300 Read block 2011 len 32768 069/300 Read block 377 len 24576 070/300 Write block 4742 len 12288 data: 4 3 2 071/300 Read block 3716 len 20480 072/300 Read block 4374 len 24576 073/300 Write block 3115 len 40960 data: 5 3 3 3 3 3 3 3 3 3 074/300 Read block 4762 len 20480 075/300 Flush 076/300 Read block 1870 len 12288 077/300 Read block 1141 len 12288 078/300 Read block 1751 len 20480 079/300 Read block 1541 len 28672 080/300 Read block 4596 len 36864 081/300 Write block 2690 len 12288 data: 5 5 5 082/300 Write block 286 len 16384 data: 4 4 3 3 083/300 Read block 3638 len 16384 084/300 Read block 736 len 8192 085/300 Read block 2245 len 20480 086/300 Write block 3103 len 32768 data: 4 4 4 6 5 4 6 5 087/300 Write block 165 len 8192 data: 3 3 088/300 Flush 089/300 Read block 4687 len 40960 090/300 Write block 4992 len 20480 data: 3 2 2 2 2 091/300 Flush 092/300 Write block 1687 len 20480 data: 5 5 3 3 3 093/300 Read block 4566 len 4096 094/300 Write block 2650 len 36864 data: 5 7 6 6 6 6 5 4 3 095/300 Read block 1943 len 8192 096/300 Read block 3388 len 40960 097/300 Read block 2022 len 8192 098/300 Write block 2305 len 40960 data: 5 5 5 5 5 5 4 5 4 4 099/300 Read block 4353 len 16384 100/300 Read block 1752 len 4096 101/300 Read block 955 len 4096 102/300 Read block 4865 len 8192 103/300 Read block 2702 len 28672 104/300 Read block 711 len 28672 105/300 Write block 4570 len 12288 data: 5 4 4 106/300 Flush 107/300 Write block 4444 len 4096 data: 6 108/300 Read block 3346 len 36864 109/300 Flush 110/300 Read block 3973 len 20480 111/300 Read block 4458 len 12288 112/300 Read block 601 len 28672 113/300 Read block 4085 len 12288 114/300 Read block 2679 len 28672 115/300 Write block 1901 len 32768 data: 4 4 4 4 3 3 3 3 116/300 Write block 976 len 28672 data: 5 4 3 3 3 3 3 117/300 Read block 3460 len 8192 118/300 Flush 119/300 Write block 616 len 8192 data: 3 3 120/300 Write block 3358 len 20480 data: 5 7 7 5 5 121/300 Write block 2486 len 32768 data: 3 4 5 5 5 5 5 5 122/300 Write block 734 len 20480 data: 4 3 4 3 3 123/300 Read block 2702 len 24576 124/300 Read block 3908 len 36864 125/300 Write block 3594 len 4096 data: 5 126/300 Read block 4940 len 20480 127/300 Read block 4478 len 12288 128/300 Flush 129/300 Read block 499 len 32768 130/300 Write block 1890 len 20480 data: 5 4 4 3 3 131/300 Read block 4962 len 28672 132/300 Write block 4453 len 28672 data: 2 2 2 2 3 3 3 133/300 Write block 2688 len 24576 data: 6 5 6 6 6 4 134/300 Read block 3010 len 32768 135/300 Write block 3074 len 24576 data: 5 5 4 6 5 4 136/300 Read block 4854 len 8192 137/300 Write block 4813 len 4096 data: 2 138/300 Read block 3998 len 24576 139/300 Read block 4303 len 32768 140/300 Read block 768 len 4096 141/300 Read block 2145 len 40960 142/300 Write block 4786 len 32768 data: 4 5 4 3 3 3 2 2 143/300 Read block 3672 len 20480 144/300 Write block 2837 len 36864 data: 3 4 4 4 3 4 6 6 6 145/300 Read block 246 len 20480 146/300 Read block 4934 len 8192 147/300 Read block 2476 len 12288 148/300 Flush 149/300 Read block 408 len 40960 150/300 Read block 2026 len 20480 151/300 Read block 4864 len 28672 152/300 Write block 2051 len 8192 data: 2 2 153/300 Read block 2042 len 20480 154/300 Write block 2760 len 40960 data: 3 3 3 3 5 6 5 5 6 8 155/300 Flush 156/300 Read block 2989 len 32768 157/300 Read block 1970 len 20480 158/300 Read block 2105 len 4096 159/300 Flush 160/300 Read block 1910 len 4096 161/300 Read block 228 len 8192 162/300 Read block 2199 len 12288 163/300 Read block 1977 len 24576 164/300 Write block 3765 len 32768 data: 7 9 7 7 5 5 5 3 165/300 Read block 552 len 24576 166/300 Read block 2074 len 16384 167/300 Write block 4881 len 12288 data: 7 7 6 168/300 Read block 4379 len 24576 169/300 Read block 4025 len 4096 170/300 Flush 171/300 Write block 3697 len 32768 data: 4 4 4 4 4 4 3 2 172/300 Write block 3432 len 4096 data: 3 173/300 Read block 4588 len 40960 174/300 Write block 3608 len 32768 data: 5 6 6 7 4 4 4 4 175/300 Write block 878 len 4096 data: 2 176/300 Read block 4201 len 8192 177/300 Read block 2130 len 12288 178/300 Read block 4838 len 8192 179/300 Flush 180/300 Read block 1281 len 8192 181/300 Read block 2760 len 8192 182/300 Flush 183/300 Read block 2921 len 12288 184/300 Read block 1625 len 24576 185/300 Read block 3757 len 12288 186/300 Read block 2927 len 4096 187/300 Write block 207 len 24576 data: 4 5 6 4 4 4 188/300 Write block 1102 len 24576 data: 5 6 6 6 6 6 189/300 Write block 2628 len 20480 data: 8 7 6 4 4 190/300 Read block 4175 len 16384 191/300 Flush 192/300 Read block 1540 len 40960 193/300 Read block 1311 len 20480 194/300 Write block 3062 len 12288 data: 5 4 4 195/300 Read block 2676 len 20480 196/300 Read block 298 len 8192 197/300 Write block 1074 len 4096 data: 6 198/300 Read block 4002 len 4096 199/300 Write block 2259 len 32768 data: 6 6 6 7 5 5 4 4 200/300 Read block 1978 len 8192 201/300 Read block 3179 len 40960 202/300 Read block 1978 len 12288 203/300 Write block 3412 len 16384 data: 4 4 4 4 204/300 Read block 2764 len 24576 205/300 Write block 2899 len 4096 data: 5 206/300 Write block 3076 len 8192 data: 5 7 207/300 Write block 6 len 32768 data: 4 3 4 5 7 9 7 7 208/300 Read block 234 len 20480 209/300 Flush 210/300 Read block 3551 len 8192 211/300 Write block 1941 len 36864 data: 2 2 3 3 3 4 4 4 4 212/300 Write block 2030 len 36864 data: 3 4 4 4 3 3 4 4 4 213/300 Read block 2774 len 28672 214/300 Read block 628 len 32768 215/300 Read block 1334 len 20480 216/300 Flush 217/300 Read block 3779 len 32768 218/300 Read block 3193 len 36864 219/300 Write block 2190 len 24576 data: 3 3 4 3 4 4 220/300 Flush 221/300 Read block 2068 len 16384 222/300 Write block 4706 len 24576 data: 3 2 2 3 4 3 223/300 Write block 2823 len 20480 data: 8 7 8 8 7 224/300 Write block 1818 len 36864 data: 3 3 3 3 3 4 4 4 4 225/300 Flush 226/300 Read block 1777 len 4096 227/300 Write block 4180 len 28672 data: 5 5 5 5 4 2 3 228/300 Write block 4264 len 40960 data: 4 4 6 5 8 8 6 5 5 4 229/300 Write block 659 len 12288 data: 7 8 7 230/300 Read block 3892 len 32768 231/300 Read block 2745 len 28672 232/300 Write block 1779 len 4096 data: 4 233/300 Flush 234/300 Write block 4008 len 12288 data: 5 7 8 235/300 Write block 2518 len 20480 data: 6 4 3 3 2 236/300 Write block 953 len 4096 data: 5 237/300 Read block 1722 len 12288 238/300 Write block 3722 len 8192 data: 3 6 239/300 Write block 293 len 24576 data: 4 5 5 6 5 4 240/300 Read block 4429 len 24576 241/300 Flush 242/300 Read block 2689 len 36864 243/300 Write block 245 len 16384 data: 4 3 3 4 244/300 Write block 2409 len 28672 data: 5 5 6 7 8 8 8 245/300 Write block 1676 len 4096 data: 6 246/300 Read block 2327 len 32768 247/300 Read block 430 len 8192 248/300 Write block 1261 len 28672 data: 7 5 6 5 5 4 3 249/300 Write block 4603 len 40960 data: 5 5 6 6 5 6 7 7 8 8 250/300 Write block 4294 len 16384 data: 5 6 4 3 251/300 Read block 2645 len 20480 252/300 Read block 2978 len 4096 253/300 Read block 2258 len 24576 254/300 Read block 2 len 8192 255/300 Read block 1243 len 12288 256/300 Read block 4548 len 20480 257/300 Read block 4970 len 16384 258/300 Write block 2051 len 40960 data: 3 3 5 5 5 6 6 4 4 4 259/300 Write block 4757 len 12288 data: 5 4 5 260/300 Read block 1582 len 36864 261/300 Read block 1294 len 20480 262/300 Write block 1754 len 40960 data: 3 6 6 6 6 8 7 5 7 5 263/300 Read block 2881 len 16384 264/300 Read block 1776 len 32768 265/300 Read block 432 len 28672 266/300 Read block 1481 len 40960 267/300 Write block 2753 len 16384 data: 4 4 4 3 268/300 Flush 269/300 Read block 2073 len 36864 270/300 Read block 2342 len 20480 271/300 Write block 2950 len 20480 data: 8 8 6 7 6 272/300 Write block 1779 len 12288 data: 5 4 4 273/300 Read block 4652 len 8192 274/300 Write block 1082 len 8192 data: 5 4 275/300 Read block 958 len 20480 276/300 Read block 4790 len 28672 277/300 Read block 2480 len 20480 278/300 Write block 2385 len 28672 data: 4 4 5 5 5 5 6 279/300 Write block 2126 len 12288 data: 3 3 3 280/300 Read block 1883 len 28672 281/300 Flush 282/300 Read block 3511 len 8192 283/300 Read block 3504 len 28672 284/300 Read block 366 len 24576 285/300 Write block 3267 len 8192 data: 5 5 286/300 Read block 2616 len 40960 287/300 Flush 288/300 Write block 2030 len 16384 data: 4 5 5 5 289/300 Read block 4505 len 12288 290/300 Flush 291/300 Read block 83 len 20480 292/300 Write block 2295 len 24576 data: 2 3 3 4 4 5 293/300 Read block 1066 len 36864 294/300 Write block 4822 len 24576 data: 4 5 4 2 3 3 295/300 Read block 2756 len 24576 296/300 Write block 80 len 4096 data: 5 297/300 Read block 2835 len 20480 298/300 Write block 2797 len 28672 data: 3 3 5 5 5 5 5 299/300 Read block 4367 len 16384 300/300 Write block 3399 len 16384 data: 3 3 3 3 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:33:43.675 INFO request completed, latency_us: 317, response_code: 204, uri: /start/cid/0, method: POST, req_id: 018a125c-024d-4686-be62-e8a8fdbcc0dc, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 4533 Acked 5532 Read 10 New Done Done true 4534 Acked 5533 Read 4 New Done Done false 4535 Acked 5534 Read 2 New Done Done false 4536 Acked 5535 Flush 0 New Done Done false 4537 Acked 5536 Read 10 New Done Done false 4538 Acked 5537 Flush 0 New Done Done false 4539 Acked 5538 Read 3 New Done Done false 4540 Acked 5539 Read 8 New Done Done false 4541 Acked 5540 Write 9 New Done Done false 4542 Acked 5541 Write 7 New Done Done false 4543 Acked 5542 Write 5 New Done Done false 4544 Acked 5543 Read 2 New Done Done false 4545 Acked 5544 Read 4 New Done Done false 4546 Acked 5545 Write 10 New Done Done false 4547 Acked 5546 Read 10 New Done Done false 4548 Acked 5547 Write 1 New Done Done false 4549 Acked 5548 Write 7 New Done Done false 4550 Acked 5549 Write 1 New Done Done false 4551 Acked 5550 Read 6 New Done Done false 4552 Acked 5551 Read 1 New Done Done false 4553 Acked 5552 Flush 0 New Done Done false 4554 Acked 5553 Write 1 New Done Done false 4555 Acked 5554 Write 3 New Done Done false 4556 Acked 5555 Read 6 New Done Done false 4557 Acked 5556 Write 7 New Done Done false 4558 Acked 5557 Write 5 New Done Done false 4559 Acked 5558 Read 1 New Done Done false 4560 Acked 5559 Read 10 New Done Done false 4561 Acked 5560 Write 4 New Done Done false 4562 Acked 5561 Write 3 New Done Done false 4563 Acked 5562 Flush 0 New Done Done false 4564 Acked 5563 Read 6 New Done Done false 4565 Acked 5564 Read 4 New Done Done false 4566 Acked 5565 Write 7 New Done Done false 4567 Acked 5566 Read 4 New Done Done false 4568 Acked 5567 Read 5 New Done Done false 4569 Acked 5568 Read 3 New Done Done false 4570 Acked 5569 Write 4 New Done Done false 4571 Acked 5570 Flush 0 New Done Done false 4572 Acked 5571 Write 1 New Done Done false 4573 Acked 5572 Read 4 New Done Done false 4574 Acked 5573 Write 10 New Done Done false 4575 Acked 5574 Read 2 New Done Done false 4576 Acked 5575 Write 6 New Done Done false 4577 Acked 5576 Write 10 New Done Done false 4578 Acked 5577 Read 2 New Done Done false 4579 Acked 5578 Flush 0 New Done Done false 4580 Acked 5579 Write 2 New Done Done false 4581 Acked 5580 Read 6 New Done Done false 4582 Acked 5581 Write 6 New Done Done false 4583 Acked 5582 Read 7 New Done Done false 4584 Acked 5583 Read 9 New Done Done false 4585 Acked 5584 Read 1 New Done Done false 4586 Acked 5585 Flush 0 New Done Done false 4587 Acked 5586 Write 9 New Done Done false 4588 Acked 5587 Read 6 New Done Done false 4589 Acked 5588 Write 4 New Done Done false 4590 Acked 5589 Flush 0 New Done Done false 4591 Acked 5590 Read 8 New Done Done false 4592 Acked 5591 Read 1 New Done Done false 4593 Acked 5592 Read 7 New Done Done false 4594 Acked 5593 Flush 0 New Done Done false 4595 Acked 5594 Read 4 New Done Done false 4596 Acked 5595 Read 5 New Done Done false 4597 Acked 5596 Write 6 New Done Done false 4598 Acked 5597 Flush 0 New Done Done false 4599 Acked 5598 Write 9 New Done Done false 4600 Acked 5599 Flush 0 New Done Done false 4601 Acked 5600 Read 8 New Done Done false 4602 Acked 5601 Read 6 New Done Done false 4603 Acked 5602 Write 3 New Done Done false 4604 Acked 5603 Read 5 New Done Done false 4605 Acked 5604 Read 6 New Done Done false 4606 Acked 5605 Write 10 New Done Done false 4607 Acked 5606 Read 5 New Done Done false 4608 Acked 5607 Flush 0 New Done Done false 4609 Acked 5608 Read 3 New Done Done false 4610 Acked 5609 Read 3 New Done Done false 4611 Acked 5610 Read 5 New Done Done false 4612 Acked 5611 Read 7 New Done Done false 4613 Acked 5612 Read 9 New Done Done false 4614 Acked 5613 Write 3 New Done Done false 4615 Acked 5614 Write 4 New Done Done false 4616 Acked 5615 Read 4 New Done Done false 4617 Acked 5616 Read 2 New Done Done false 4618 Acked 5617 Read 5 New Done Done false 4619 Acked 5618 Write 8 New Done Done false 4620 Acked 5619 Write 2 New Done Done false 4621 Acked 5620 Flush 0 New Done Done false 4622 Acked 5621 Read 10 New Done Done false 4623 Acked 5622 Write 5 New Done Done false 4624 Acked 5623 Flush 0 New Done Done false 4625 Acked 5624 Write 5 New Done Done false 4626 Acked 5625 Read 1 New Done Done false 4627 Acked 5626 Write 9 New Done Done false 4628 Acked 5627 Read 2 New Done Done false 4629 Acked 5628 Read 10 New Done Done false 4630 Acked 5629 Read 2 New Done Done false 4631 Acked 5630 Write 10 New Done Done false 4632 Acked 5631 Read 4 New Done Done false 4633 Acked 5632 Read 1 New Done Done false 4634 Acked 5633 Read 1 New Done Done false 4635 Acked 5634 Read 2 New Done Done false 4636 Acked 5635 Read 7 New Done Done false 4637 Acked 5636 Read 7 New Done Done false 4638 Acked 5637 Write 3 New Done Done false 4639 Acked 5638 Flush 0 New Done Done false 4640 Acked 5639 Write 1 New Done Done false 4641 Acked 5640 Read 9 New Done Done false 4642 Acked 5641 Flush 0 New Done Done false 4643 Acked 5642 Read 5 New Done Done false 4644 Acked 5643 Read 3 New Done Done false 4645 Acked 5644 Read 7 New Done Done false 4646 Acked 5645 Read 3 New Done Done false 4647 Acked 5646 Read 7 New Done Done false 4648 Acked 5647 Write 8 New Done Done false 4649 Acked 5648 Write 7 New Done Done false 4650 Acked 5649 Read 2 New Done Done false 4651 Acked 5650 Flush 0 New Done Done false 4652 Acked 5651 Write 2 New Done Done false 4653 Acked 5652 Write 5 New Done Done false 4654 Acked 5653 Write 8 New Done Done false 4655 Acked 5654 Write 5 New Done Done false 4656 Acked 5655 Read 6 New Done Done false 4657 Acked 5656 Read 9 New Done Done false 4658 Acked 5657 Write 1 New Done Done false 4659 Acked 5658 Read 5 New Done Done false 4660 Acked 5659 Read 3 New Done Done false 4661 Acked 5660 Flush 0 New Done Done false 4662 Acked 5661 Read 8 New Done Done false 4663 Acked 5662 Write 5 New Done Done false 4664 Acked 5663 Read 7 New Done Done false 4665 Acked 5664 Write 7 New Done Done false 4666 Acked 5665 Write 6 New Done Done false 4667 Acked 5666 Read 8 New Done Done false 4668 Acked 5667 Write 6 New Done Done false 4669 Acked 5668 Read 2 New Done Done false 4670 Acked 5669 Write 1 New Done Done false 4671 Acked 5670 Read 6 New Done Done false 4672 Acked 5671 Read 8 New Done Done false 4673 Acked 5672 Read 1 New Done Done false 4674 Acked 5673 Read 10 New Done Done false 4675 Acked 5674 Write 8 New Done Done false 4676 Acked 5675 Read 5 New Done Done false 4677 Acked 5676 Write 9 New Done Done false 4678 Acked 5677 Read 5 New Done Done false 4679 Acked 5678 Read 2 New Done Done false 4680 Acked 5679 Read 3 New Done Done false 4681 Acked 5680 Flush 0 New Done Done false 4682 Acked 5681 Read 10 New Done Done false 4683 Acked 5682 Read 5 New Done Done false 4684 Acked 5683 Read 7 New Done Done false 4685 Acked 5684 Write 2 New Done Done false 4686 Acked 5685 Read 5 New Done Done false 4687 Acked 5686 Write 10 New Done Done false 4688 Acked 5687 Flush 0 New Done Done false 4689 Acked 5688 Read 8 New Done Done false 4690 Acked 5689 Read 5 New Done Done false 4691 Acked 5690 Read 1 New Done Done false 4692 Acked 5691 Flush 0 New Done Done false 4693 Acked 5692 Read 1 New Done Done false 4694 Acked 5693 Read 2 New Done Done false 4695 Acked 5694 Read 3 New Done Done false 4696 Acked 5695 Read 6 New Done Done false 4697 Acked 5696 Write 8 New Done Done false 4698 Acked 5697 Read 6 New Done Done false 4699 Acked 5698 Read 4 New Done Done false 4700 Acked 5699 Write 3 New Done Done false 4701 Acked 5700 Read 6 New Done Done false 4702 Acked 5701 Read 1 New Done Done false 4703 Acked 5702 Flush 0 New Done Done false 4704 Acked 5703 Write 8 New Done Done false 4705 Acked 5704 Write 1 New Done Done false 4706 Acked 5705 Read 10 New Done Done false 4707 Acked 5706 Write 8 New Done Done false 4708 Acked 5707 Write 1 New Done Done false 4709 Acked 5708 Read 2 New Done Done false 4710 Acked 5709 Read 3 New Done Done false 4711 Acked 5710 Read 2 New Done Done false 4712 Acked 5711 Flush 0 New Done Done false 4713 Acked 5712 Read 2 New Done Done false 4714 Acked 5713 Read 2 New Done Done false 4715 Acked 5714 Flush 0 New Done Done false 4716 Acked 5715 Read 3 New Done Done false 4717 Acked 5716 Read 6 New Done Done false 4718 Acked 5717 Read 3 New Done Done false 4719 Acked 5718 Read 1 New Done Done false 4720 Acked 5719 Write 6 New Done Done false 4721 Acked 5720 Write 6 New Done Done false 4722 Acked 5721 Write 5 New Done Done false 4723 Acked 5722 Read 4 New Done Done false 4724 Acked 5723 Flush 0 New Done Done false 4725 Acked 5724 Read 10 New Done Done false 4726 Acked 5725 Read 5 New Done Done false 4727 Acked 5726 Write 3 New Done Done false 4728 Acked 5727 Read 5 New Done Done false 4729 Acked 5728 Read 2 New Done Done false 4730 Acked 5729 Write 1 New Done Done false 4731 Acked 5730 Read 1 New Done Done false 4732 Acked 5731 Write 8 New Done Done false 4733 Acked 5732 Read 2 New Done Done false 4734 Acked 5733 Read 10 New Done Done false 4735 Acked 5734 Read 3 New Done Done false 4736 Acked 5735 Write 4 New Done Done false 4737 Acked 5736 Read 6 New Done Done false 4738 Acked 5737 Write 1 New Done Done false 4739 Acked 5738 Write 2 New Done Done false 4740 Acked 5739 Write 8 New Done Done false 4741 Acked 5740 Read 5 New Done Done false 4742 Acked 5741 Flush 0 New Done Done false 4743 Acked 5742 Read 2 New Done Done false 4744 Acked 5743 Write 9 New Done Done false 4745 Acked 5744 Write 9 New Done Done false 4746 Acked 5745 Read 7 New Done Done false 4747 Acked 5746 Read 8 New Done Done false 4748 Acked 5747 Read 5 New Done Done false 4749 Acked 5748 Flush 0 New Done Done false 4750 Acked 5749 Read 8 New Done Done false 4751 Acked 5750 Read 9 New Done Done false 4752 Acked 5751 Write 6 New Done Done false 4753 Acked 5752 Flush 0 New Done Done false 4754 Acked 5753 Read 4 New Done Done false 4755 Acked 5754 Write 6 New Done Done false 4756 Acked 5755 Write 5 New Done Done false 4757 Acked 5756 Write 9 New Done Done false 4758 Acked 5757 Flush 0 New Done Done false 4759 Acked 5758 Read 1 New Done Done false 4760 Acked 5759 Write 7 New Done Done false 4761 Acked 5760 Write 10 New Done Done false 4762 Acked 5761 Write 3 New Done Done false 4763 Acked 5762 Read 8 New Done Done false 4764 Acked 5763 Read 7 New Done Done false 4765 Acked 5764 Write 1 New Done Done false 4766 Acked 5765 Flush 0 New Done Done false 4767 Acked 5766 Write 3 New Done Done false 4768 Acked 5767 Write 5 New Done Done false 4769 Acked 5768 Write 1 New Done Done false 4770 Acked 5769 Read 3 New Done Done false 4771 Acked 5770 Write 2 New Done Done false 4772 Acked 5771 Write 6 New Done Done false 4773 Acked 5772 Read 6 New Done Done false 4774 Acked 5773 Flush 0 New Done Done false 4775 Acked 5774 Read 9 New Done Done false 4776 Acked 5775 Write 4 New Done Done false 4777 Acked 5776 Write 7 New Done Done false 4778 Acked 5777 Write 1 New Done Done false 4779 Acked 5778 Read 8 New Done Done false 4780 Acked 5779 Read 2 New Done Done false 4781 Acked 5780 Write 7 New Done Done false 4782 Acked 5781 Write 10 New Done Done false 4783 Acked 5782 Write 4 New Done Done false 4784 Acked 5783 Read 5 New Done Done false 4785 Acked 5784 Read 1 New Done Done false 4786 Acked 5785 Read 6 New Done Done false 4787 Acked 5786 Read 2 New Done Done false 4788 Acked 5787 Read 3 New Done Done false 4789 Acked 5788 Read 5 New Done Done false 4790 Acked 5789 Read 4 New Done Done false 4791 Acked 5790 Write 10 New Done Done false 4792 Acked 5791 Write 3 New Done Done false 4793 Acked 5792 Read 9 New Done Done false 4794 Acked 5793 Read 5 New Done Done false 4795 Acked 5794 Write 10 New Done Done false 4796 Acked 5795 Read 4 New Done Done false 4797 Acked 5796 Read 8 New Done Done false 4798 Acked 5797 Read 7 New Done Done false 4799 Acked 5798 Read 10 New Done Done false 4800 Acked 5799 Write 4 New Done Done false 4801 Acked 5800 Flush 0 New Done Done false 4802 Acked 5801 Read 9 New Done Done false 4803 Acked 5802 Read 5 New Done Done false 4804 Acked 5803 Write 5 New Done Done false 4805 Acked 5804 Write 3 New Done Done false 4806 Acked 5805 Read 2 New Done Done false 4807 Acked 5806 Write 2 New Done Done false 4808 Acked 5807 Read 5 New Done Done false 4809 Acked 5808 Read 7 New Done Done false 4810 Acked 5809 Read 5 New Done Done false 4811 Acked 5810 Write 7 New Done Done false 4812 Acked 5811 Write 3 New Done Done false 4813 Acked 5812 Read 7 New Done Done false 4814 Acked 5813 Flush 0 New Done Done false 4815 Acked 5814 Read 2 New Done Done false 4816 Acked 5815 Read 7 New Done Done false 4817 Acked 5816 Read 6 New Done Done false 4818 Acked 5817 Write 2 New Done Done false 4819 Acked 5818 Read 10 New Done Done false 4820 Acked 5819 Flush 0 New Done Done false 4821 Acked 5820 Write 4 New Done Done false 4822 Acked 5821 Read 3 New Done Done false 4823 Acked 5822 Flush 0 New Done Done false 4824 Acked 5823 Read 5 New Done Done false 4825 Acked 5824 Write 6 New Done Done false 4826 Acked 5825 Read 9 New Done Done false 4827 Acked 5826 Write 6 New Done Done false 4828 Acked 5827 Read 6 New Done Done false 4829 Acked 5828 Write 1 New Done Done false 4830 Acked 5829 Read 5 New Done Done false 4831 Acked 5830 Write 7 New Done Done false 4832 Acked 5831 Read 4 New Done Done false 4833 Acked 5832 Write 4 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5531 5822 5822 Downstairs last five completed: 5531 5530 5529 5528 5527 Upstairs last five completed: 4833 4832 4831 4830 4829 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:44.090979731Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:44.091092918Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:44.091157826Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:44.091184644Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:44.091205145Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:44.091249121Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:44.091507612Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:45.092253679Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:45.092326695Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:45.092355275Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:45.092382032Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:45.092402456Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:45.092514842Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:45.092781095Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:46.093765259Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:46.093879445Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:46.093907596Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:46.093934223Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:46.093954593Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:46.094018457Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:46.094276837Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:47.095141078Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:47.095207445Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:47.095235961Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:47.095260126Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:47.095286296Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:47.095306663Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:47.095337085Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:47.095592138Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1201) [8810][0] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 4533 Acked 5532 Read 10 New Done Done true 4534 Acked 5533 Read 4 New Done Done true 4535 Acked 5534 Read 2 New Done Done true 4536 Acked 5535 Flush 0 New Done Done true 4537 Acked 5536 Read 10 New Done Done true 4538 Acked 5537 Flush 0 New Done Done true 4539 Acked 5538 Read 3 New Done Done true 4540 Acked 5539 Read 8 New Done Done true 4541 Acked 5540 Write 9 New Done Done true 4542 Acked 5541 Write 7 New Done Done true 4543 Acked 5542 Write 5 New Done Done true 4544 Acked 5543 Read 2 New Done Done true 4545 Acked 5544 Read 4 New Done Done true 4546 Acked 5545 Write 10 New Done Done true 4547 Acked 5546 Read 10 New Done Done true 4548 Acked 5547 Write 1 New Done Done true 4549 Acked 5548 Write 7 New Done Done true 4550 Acked 5549 Write 1 New Done Done true 4551 Acked 5550 Read 6 New Done Done true 4552 Acked 5551 Read 1 New Done Done true 4553 Acked 5552 Flush 0 New Done Done true 4554 Acked 5553 Write 1 New Done Done true 4555 Acked 5554 Write 3 New Done Done true 4556 Acked 5555 Read 6 New Done Done true 4557 Acked 5556 Write 7 New Done Done true 4558 Acked 5557 Write 5 New Done Done true 4559 Acked 5558 Read 1 New Done Done true 4560 Acked 5559 Read 10 New Done Done true 4561 Acked 5560 Write 4 New Done Done true 4562 Acked 5561 Write 3 New Done Done true 4563 Acked 5562 Flush 0 New Done Done true 4564 Acked 5563 Read 6 New Done Done true 4565 Acked 5564 Read 4 New Done Done true 4566 Acked 5565 Write 7 New Done Done true 4567 Acked 5566 Read 4 New Done Done true 4568 Acked 5567 Read 5 New Done Done true 4569 Acked 5568 Read 3 New Done Done true 4570 Acked 5569 Write 4 New Done Done true 4571 Acked 5570 Flush 0 New Done Done true 4572 Acked 5571 Write 1 New Done Done true 4573 Acked 5572 Read 4 New Done Done true 4574 Acked 5573 Write 10 New Done Done true 4575 Acked 5574 Read 2 New Done Done true 4576 Acked 5575 Write 6 New Done Done true 4577 Acked 5576 Write 10 New Done Done true 4578 Acked 5577 Read 2 New Done Done true 4579 Acked 5578 Flush 0 New Done Done true 4580 Acked 5579 Write 2 New Done Done true 4581 Acked 5580 Read 6 New Done Done true 4582 Acked 5581 Write 6 New Done Done true 4583 Acked 5582 Read 7 New Done Done true 4584 Acked 5583 Read 9 New Done Done true 4585 Acked 5584 Read 1 New Done Done true 4586 Acked 5585 Flush 0 New Done Done true 4587 Acked 5586 Write 9 New Done Done true 4588 Acked 5587 Read 6 New Done Done true 4589 Acked 5588 Write 4 New Done Done true 4590 Acked 5589 Flush 0 New Done Done true 4591 Acked 5590 Read 8 New Done Done true 4592 Acked 5591 Read 1 New Done Done true 4593 Acked 5592 Read 7 New Done Done true 4594 Acked 5593 Flush 0 New Done Done true 4595 Acked 5594 Read 4 New Done Done true 4596 Acked 5595 Read 5 New Done Done true 4597 Acked 5596 Write 6 New Done Done true 4598 Acked 5597 Flush 0 New Done Done true 4599 Acked 5598 Write 9 New Done Done true 4600 Acked 5599 Flush 0 New Done Done true 4601 Acked 5600 Read 8 New Done Done true 4602 Acked 5601 Read 6 New Done Done true 4603 Acked 5602 Write 3 New Done Done true 4604 Acked 5603 Read 5 New Done Done true 4605 Acked 5604 Read 6 New Done Done true 4606 Acked 5605 Write 10 New Done Done true 4607 Acked 5606 Read 5 New Done Done true 4608 Acked 5607 Flush 0 New Done Done true 4609 Acked 5608 Read 3 New Done Done true 4610 Acked 5609 Read 3 New Done Done true 4611 Acked 5610 Read 5 New Done Done true 4612 Acked 5611 Read 7 New Done Done true 4613 Acked 5612 Read 9 New Done Done true 4614 Acked 5613 Write 3 New Done Done true 4615 Acked 5614 Write 4 New Done Done true 4616 Acked 5615 Read 4 New Done Done true 4617 Acked 5616 Read 2 New Done Done true 4618 Acked 5617 Read 5 New Done Done true 4619 Acked 5618 Write 8 New Done Done true 4620 Acked 5619 Write 2 New Done Done true 4621 Acked 5620 Flush 0 New Done Done true 4622 Acked 5621 Read 10 New Done Done true 4623 Acked 5622 Write 5 New Done Done true 4624 Acked 5623 Flush 0 New Done Done true 4625 Acked 5624 Write 5 New Done Done true 4626 Acked 5625 Read 1 New Done Done true 4627 Acked 5626 Write 9 New Done Done true 4628 Acked 5627 Read 2 New Done Done true 4629 Acked 5628 Read 10 New Done Done true 4630 Acked 5629 Read 2 New Done Done true 4631 Acked 5630 Write 10 New Done Done true 4632 Acked 5631 Read 4 New Done Done true 4633 Acked 5632 Read 1 New Done Done true 4634 Acked 5633 Read 1 New Done Done true 4635 Acked 5634 Read 2 New Done Done true 4636 Acked 5635 Read 7 New Done Done true 4637 Acked 5636 Read 7 New Done Done true 4638 Acked 5637 Write 3 New Done Done true 4639 Acked 5638 Flush 0 New Done Done true 4640 Acked 5639 Write 1 New Done Done true 4641 Acked 5640 Read 9 New Done Done true 4642 Acked 5641 Flush 0 New Done Done true 4643 Acked 5642 Read 5 New Done Done true 4644 Acked 5643 Read 3 New Done Done true 4645 Acked 5644 Read 7 New Done Done true 4646 Acked 5645 Read 3 New Done Done true 4647 Acked 5646 Read 7 New Done Done true 4648 Acked 5647 Write 8 New Done Done true 4649 Acked 5648 Write 7 New Done Done true 4650 Acked 5649 Read 2 New Done Done true 4651 Acked 5650 Flush 0 New Done Done true 4652 Acked 5651 Write 2 New Done Done true 4653 Acked 5652 Write 5 New Done Done true 4654 Acked 5653 Write 8 New Done Done true 4655 Acked 5654 Write 5 New Done Done true 4656 Acked 5655 Read 6 New Done Done true 4657 Acked 5656 Read 9 New Done Done true 4658 Acked 5657 Write 1 New Done Done true 4659 Acked 5658 Read 5 New Done Done true 4660 Acked 5659 Read 3 New Done Done true 4661 Acked 5660 Flush 0 New Done Done true 4662 Acked 5661 Read 8 New Done Done true 4663 Acked 5662 Write 5 New Done Done true 4664 Acked 5663 Read 7 New Done Done true 4665 Acked 5664 Write 7 New Done Done true 4666 Acked 5665 Write 6 New Done Done true 4667 Acked 5666 Read 8 New Done Done true 4668 Acked 5667 Write 6 New Done Done true 4669 Acked 5668 Read 2 New Done Done true 4670 Acked 5669 Write 1 New Done Done true 4671 Acked 5670 Read 6 New Done Done true 4672 Acked 5671 Read 8 New Done Done true 4673 Acked 5672 Read 1 New Done Done true 4674 Acked 5673 Read 10 New Done Done true 4675 Acked 5674 Write 8 New Done Done true 4676 Acked 5675 Read 5 New Done Done true 4677 Acked 5676 Write 9 New Done Done true 4678 Acked 5677 Read 5 New Done Done true 4679 Acked 5678 Read 2 New Done Done true 4680 Acked 5679 Read 3 New Done Done true 4681 Acked 5680 Flush 0 New Done Done true 4682 Acked 5681 Read 10 New Done Done true 4683 Acked 5682 Read 5 New Done Done true 4684 Acked 5683 Read 7 New Done Done true 4685 Acked 5684 Write 2 New Done Done true 4686 Acked 5685 Read 5 New Done Done true 4687 Acked 5686 Write 10 New Done Done true 4688 Acked 5687 Flush 0 New Done Done true 4689 Acked 5688 Read 8 New Done Done true 4690 Acked 5689 Read 5 New Done Done true 4691 Acked 5690 Read 1 New Done Done true 4692 Acked 5691 Flush 0 New Done Done true 4693 Acked 5692 Read 1 New Done Done true 4694 Acked 5693 Read 2 New Done Done true 4695 Acked 5694 Read 3 New Done Done true 4696 Acked 5695 Read 6 New Done Done true 4697 Acked 5696 Write 8 New Done Done true 4698 Acked 5697 Read 6 New Done Done true 4699 Acked 5698 Read 4 New Done Done true 4700 Acked 5699 Write 3 New Done Done true 4701 Acked 5700 Read 6 New Done Done true 4702 Acked 5701 Read 1 New Done Done true 4703 Acked 5702 Flush 0 New Done Done true 4704 Acked 5703 Write 8 New Done Done true 4705 Acked 5704 Write 1 New Done Done true 4706 Acked 5705 Read 10 New Done Done true 4707 Acked 5706 Write 8 New Done Done true 4708 Acked 5707 Write 1 New Done Done true 4709 Acked 5708 Read 2 New Done Done true 4710 Acked 5709 Read 3 New Done Done true 4711 Acked 5710 Read 2 New Done Done true 4712 Acked 5711 Flush 0 New Done Done true 4713 Acked 5712 Read 2 New Done Done true 4714 Acked 5713 Read 2 New Done Done true 4715 Acked 5714 Flush 0 New Done Done true 4716 Acked 5715 Read 3 New Done Done true 4717 Acked 5716 Read 6 New Done Done true 4718 Acked 5717 Read 3 New Done Done true 4719 Acked 5718 Read 1 New Done Done true 4720 Acked 5719 Write 6 New Done Done true 4721 Acked 5720 Write 6 New Done Done true 4722 Acked 5721 Write 5 New Done Done true 4723 Acked 5722 Read 4 New Done Done true 4724 Acked 5723 Flush 0 New Done Done true 4725 Acked 5724 Read 10 New Done Done true 4726 Acked 5725 Read 5 New Done Done true 4727 Acked 5726 Write 3 New Done Done true 4728 Acked 5727 Read 5 New Done Done true 4729 Acked 5728 Read 2 New Done Done true 4730 Acked 5729 Write 1 New Done Done true 4731 Acked 5730 Read 1 New Done Done true 4732 Acked 5731 Write 8 New Done Done true 4733 Acked 5732 Read 2 New Done Done true 4734 Acked 5733 Read 10 New Done Done true 4735 Acked 5734 Read 3 New Done Done true 4736 Acked 5735 Write 4 New Done Done true 4737 Acked 5736 Read 6 New Done Done true 4738 Acked 5737 Write 1 New Done Done true 4739 Acked 5738 Write 2 New Done Done true 4740 Acked 5739 Write 8 New Done Done true 4741 Acked 5740 Read 5 New Done Done true 4742 Acked 5741 Flush 0 New Done Done true 4743 Acked 5742 Read 2 New Done Done true 4744 Acked 5743 Write 9 New Done Done true 4745 Acked 5744 Write 9 New Done Done true 4746 Acked 5745 Read 7 New Done Done true 4747 Acked 5746 Read 8 New Done Done true 4748 Acked 5747 Read 5 New Done Done true 4749 Acked 5748 Flush 0 New Done Done true 4750 Acked 5749 Read 8 New Done Done true 4751 Acked 5750 Read 9 New Done Done true 4752 Acked 5751 Write 6 New Done Done true 4753 Acked 5752 Flush 0 New Done Done true 4754 Acked 5753 Read 4 New Done Done true 4755 Acked 5754 Write 6 New Done Done true 4756 Acked 5755 Write 5 New Done Done true 4757 Acked 5756 Write 9 New Done Done true 4758 Acked 5757 Flush 0 New Done Done true 4759 Acked 5758 Read 1 New Done Done true 4760 Acked 5759 Write 7 New Done Done true 4761 Acked 5760 Write 10 New Done Done true 4762 Acked 5761 Write 3 New Done Done true 4763 Acked 5762 Read 8 New Done Done true 4764 Acked 5763 Read 7 New Done Done true 4765 Acked 5764 Write 1 New Done Done true 4766 Acked 5765 Flush 0 New Done Done true 4767 Acked 5766 Write 3 New Done Done true 4768 Acked 5767 Write 5 New Done Done true 4769 Acked 5768 Write 1 New Done Done true 4770 Acked 5769 Read 3 New Done Done true 4771 Acked 5770 Write 2 New Done Done true 4772 Acked 5771 Write 6 New Done Done true 4773 Acked 5772 Read 6 New Done Done true 4774 Acked 5773 Flush 0 New Done Done true 4775 Acked 5774 Read 9 New Done Done true 4776 Acked 5775 Write 4 New Done Done true 4777 Acked 5776 Write 7 New Done Done true 4778 Acked 5777 Write 1 New Done Done true 4779 Acked 5778 Read 8 New Done Done true 4780 Acked 5779 Read 2 New Done Done true 4781 Acked 5780 Write 7 New Done Done true 4782 Acked 5781 Write 10 New Done Done true 4783 Acked 5782 Write 4 New Done Done true 4784 Acked 5783 Read 5 New Done Done true 4785 Acked 5784 Read 1 New Done Done true 4786 Acked 5785 Read 6 New Done Done true 4787 Acked 5786 Read 2 New Done Done true 4788 Acked 5787 Read 3 New Done Done true 4789 Acked 5788 Read 5 New Done Done true 4790 Acked 5789 Read 4 New Done Done true 4791 Acked 5790 Write 10 New Done Done true 4792 Acked 5791 Write 3 New Done Done true 4793 Acked 5792 Read 9 New Done Done true 4794 Acked 5793 Read 5 New Done Done true 4795 Acked 5794 Write 10 New Done Done true 4796 Acked 5795 Read 4 New Done Done true 4797 Acked 5796 Read 8 New Done Done true 4798 Acked 5797 Read 7 New Done Done true 4799 Acked 5798 Read 10 New Done Done true 4800 Acked 5799 Write 4 New Done Done true 4801 Acked 5800 Flush 0 New Done Done true 4802 Acked 5801 Read 9 New Done Done true 4803 Acked 5802 Read 5 New Done Done true 4804 Acked 5803 Write 5 New Done Done true 4805 Acked 5804 Write 3 New Done Done true 4806 Acked 5805 Read 2 New Done Done true 4807 Acked 5806 Write 2 New Done Done true 4808 Acked 5807 Read 5 New Done Done true 4809 Acked 5808 Read 7 New Done Done true 4810 Acked 5809 Read 5 New Done Done true 4811 Acked 5810 Write 7 New Done Done true 4812 Acked 5811 Write 3 New Done Done true 4813 Acked 5812 Read 7 New Done Done true 4814 Acked 5813 Flush 0 New Done Done true 4815 Acked 5814 Read 2 New Done Done true 4816 Acked 5815 Read 7 New Done Done true 4817 Acked 5816 Read 6 New Done Done true 4818 Acked 5817 Write 2 New Done Done true 4819 Acked 5818 Read 10 New Done Done true 4820 Acked 5819 Flush 0 New Done Done true 4821 Acked 5820 Write 4 New Done Done true 4822 Acked 5821 Read 3 New Done Done true 4823 Acked 5822 Flush 0 New Done Done true 4824 Acked 5823 Read 5 New Done Done true 4825 Acked 5824 Write 6 New Done Done true 4826 Acked 5825 Read 9 New Done Done true 4827 Acked 5826 Write 6 New Done Done true 4828 Acked 5827 Read 6 New Done Done true 4829 Acked 5828 Write 1 New Done Done true 4830 Acked 5829 Read 5 New Done Done true 4831 Acked 5830 Write 7 New Done Done true 4832 Acked 5831 Read 4 New Done Done true 4833 Acked 5832 Write 4 New Done Done true 4834 Acked 5833 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5531 5833 5833 Downstairs last five completed: 5531 5530 5529 5528 5527 Upstairs last five completed: 4834 4833 4832 4831 4830 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.096280376Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.0964287Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098437631Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098475712Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098499814Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 5531","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098786227Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098827Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098868585Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098911268Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:48.098953908Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 5833 5832 5831 5830 5829 Upstairs last five completed: 4834 4833 4832 4831 4830 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:33:51.681 INFO request completed, latency_us: 456, response_code: 204, uri: /stop/cid/0, method: POST, req_id: a26ad13d-5a02-420c-aab6-51c424e2d713, remote_addr: 127[8810] Got stop action so:false kr:false .0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:33:51.682 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0915bf9b-b8e8-4119-a81a-06ae4dc16031, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:51.682514623Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:51.682664763Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:51.68270815Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:51.682731226Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:51.682754213Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:51.682830524Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:52.683205203Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:52.683296867Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:52.683324962Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:52.683351469Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:52.683382529Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:52.68340628Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:52.683438383Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:53.684466624Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:53.684574007Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:53.684634333Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:53.684680922Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:53.684701131Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:53.684731937Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:53.684788206Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:54.685639451Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:54.685708168Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:54.685751872Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:54.685782697Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:54.685814063Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:54.685837531Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:54.685859704Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:33:55.685 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/0, method: GET, req_id: d2dce164-8ad9-4be2-83cb-2e52c90e5763, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:55.68607605Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:55.686139069Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:55.686167432Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:55.686193873Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:55.68621422Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:55.686245598Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:55.68635494Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 001/300 Read block 2689 len 4096 002/300 Read block 1408 len 16384 003/300 Write block 746 len 8192 data: 3 3 004/300 Read block 697 len 36864 005/300 Read block 3862 len 28672 006/300 Read block 2945 len 20480 007/300 Write block 4657 len 28672 data: 5 5 2 2 3 3 3 008/300 Read block 3816 len 12288 009/300 Read block 3486 len 8192 010/300 Read block 2503 len 16384 011/300 Write block 3257 len 32768 data: 6 5 7 6 5 5 5 5 012/300 Read block 4519 len 4096 013/300 Read block 1068 len 24576 014/300 Write block 1286 len 28672 data: 4 4 6 4 4 3 2 015/300 Flush 016/300 Write block 4050 len 16384 data: 4 3 3 2 017/300 Read block 154 len 24576 018/300 Flush 019/300 Read block 184 len 36864 020/300 Write block 1443 len 12288 data: 3 3 3 021/300 Read block 152 len 12288 022/300 Flush 023/300 Read block 2090 len 28672 024/300 Write block 88 len 24576 data: 6 4 4 4 4 5 025/300 Read block 4172 len 40960 026/300 Read block 3681 len 36864 027/300 Write block 2815 len 8192 data: 2 5 028/300 Read block 4650 len 16384 029/300 Flush 030/300 Read block 748 len 24576 031/300 Write block 1651 len 16384 data: 3 3 4 3 032/300 Read block 1550 len 20480 033/300 Read block 4229 len 36864 034/300 Flush 035/300 Write block 1286 len 4096 data: 5 036/300 Write block 1700 len 32768 data: 3 3 3 3 4 5 5 4 037/300 Read block 74 len 12288 038/300 Write block 1383 len 40960 data: 2 2 2 2 3 2 5 5 6 9 039/300 Write block 4154 len 40960 data: 4 4 4 4 2 2 2 2 3 3 040/300 Read block 3444 len 36864 041/300 Write block 878 len 12288 data: 3 2 3 042/300 Flush 043/300 Write block 1176 len 28672 data: 3 3 2 2 2 3 2 044/300 Write block 725 len 8192 data: 4 4 045/300 Flush 046/300 Read block 3913 len 40960 047/300 Write block 145 len 28672 data: 4 5 4 5 5 4 3 048/300 Write block 1805 len 24576 data: 2 2 2 2 2 3 049/300 Read block 4975 len 24576 050/300 Read block 2372 len 8192 051/300 Write block 2407 len 12288 data: 4 4 6 052/300 Write block 3786 len 8192 data: 2 3 053/300 Write block 4082 len 16384 data: 6 5 5 5 054/300 Read block 1680 len 20480 055/300 Write block 998 len 16384 data: 3 3 3 4 056/300 Read block 3095 len 40960 057/300 Write block 1206 len 36864 data: 4 4 4 3 3 3 3 3 3 058/300 Write block 1789 len 16384 data: 3 3 3 3 059/300 Write block 318 len 4096 data: 7 060/300 Read block 3997 len 24576 061/300 Read block 3411 len 16384 062/300 Read block 4228 len 24576 063/300 Read block 701 len 24576 064/300 Read block 1378 len 40960 065/300 Read block 2773 len 36864 066/300 Write block 624 len 32768 data: 2 2 2 2 2 2 2 2 067/300 Write block 891 len 28672 data: 5 6 5 5 5 5 6 068/300 Flush 069/300 Write block 3791 len 4096 data: 4 070/300 Read block 3764 len 24576 071/300 Read block 3770 len 8192 072/300 Write block 3951 len 4096 data: 3 073/300 Read block 226 len 28672 074/300 Flush 075/300 Read block 290 len 16384 076/300 Write block 3347 len 16384 data: 6 6 5 5 077/300 Read block 3584 len 24576 078/300 Read block 2301 len 24576 079/300 Write block 2083 len 8192 data: 2 2 080/300 Read block 4525 len 4096 081/300 Write block 1426 len 16384 data: 3 3 4 4 082/300 Write block 3634 len 16384 data: 5 4 4 5 083/300 Read block 1194 len 24576 084/300 Write block 577 len 36864 data: 4 4 4 4 4 4 5 4 4 085/300 Read block 4664 len 12288 086/300 Write block 2597 len 16384 data: 3 3 3 2 087/300 Write block 4283 len 16384 data: 3 3 3 3 088/300 Read block 3053 len 40960 089/300 Read block 1746 len 8192 090/300 Read block 263 len 28672 091/300 Read block 3395 len 28672 092/300 Read block 272 len 24576 093/300 Write block 662 len 36864 data: 7 9 9 9 7 7 4 4 5 094/300 Write block 1621 len 28672 data: 2 2 2 4 7 6 6 095/300 Write block 4927 len 28672 data: 7 7 5 4 6 5 6 096/300 Read block 2626 len 40960 097/300 Write block 2522 len 20480 data: 3 2 2 2 3 098/300 Read block 4348 len 20480 099/300 Write block 2306 len 16384 data: 6 6 6 6 100/300 Read block 711 len 24576 101/300 Write block 4265 len 24576 data: 5 7 6 9 9 7 102/300 Write block 2064 len 20480 data: 3 2 2 2 2 103/300 Read block 597 len 12288 104/300 Read block 2329 len 12288 105/300 Read block 2976 len 32768 106/300 Write block 1920 len 4096 data: 4 107/300 Write block 2886 len 20480 data: 3 3 4 4 4 108/300 Flush 109/300 Read block 2014 len 12288 110/300 Read block 563 len 12288 111/300 Flush 112/300 Write block 2776 len 4096 data: 4 113/300 Read block 2838 len 20480 114/300 Read block 2073 len 16384 115/300 Write block 398 len 32768 data: 5 6 5 5 4 4 4 2 116/300 Flush 117/300 Read block 1490 len 36864 118/300 Write block 1529 len 16384 data: 2 2 2 2 119/300 Flush 120/300 Flush 121/300 Read block 2563 len 16384 122/300 Flush 123/300 Write block 4148 len 4096 data: 6 124/300 Read block 4662 len 28672 125/300 Read block 4818 len 24576 126/300 Read block 3198 len 12288 127/300 Read block 230 len 16384 128/300 Read block 2019 len 8192 129/300 Read block 441 len 4096 130/300 Read block 3315 len 36864 131/300 Read block 2205 len 24576 132/300 Read block 377 len 28672 133/300 Read block 1400 len 40960 134/300 Read block 755 len 8192 135/300 Read block 3128 len 32768 136/300 Read block 3793 len 16384 137/300 Write block 4822 len 12288 data: 5 6 5 138/300 Read block 2735 len 16384 139/300 Write block 1461 len 12288 data: 3 4 4 140/300 Read block 2662 len 20480 141/300 Read block 3459 len 8192 142/300 Read block 1110 len 24576 143/300 Read block 3279 len 16384 144/300 Read block 2396 len 4096 145/300 Read block 3084 len 8192 146/300 Write block 3594 len 32768 data: 6 4 3 3 4 3 4 4 147/300 Write block 3445 len 20480 data: 2 2 2 2 2 148/300 Write block 3004 len 36864 data: 3 3 2 2 3 3 4 3 4 149/300 Read block 3111 len 24576 150/300 Read block 1120 len 36864 151/300 Write block 242 len 36864 data: 3 3 3 5 4 4 5 3 3 152/300 Write block 4935 len 36864 data: 8 7 5 6 4 3 3 3 3 153/300 Read block 2067 len 24576 154/300 Write block 2894 len 32768 data: 5 5 6 6 7 6 5 5 155/300 Read block 3232 len 32768 156/300 Read block 4648 len 24576 157/300 Read block 202 len 32768 158/300 Write block 486 len 40960 data: 2 2 2 2 3 3 3 3 3 3 159/300 Write block 2414 len 20480 data: 9 9 8 8 8 160/300 Read block 1764 len 20480 161/300 Read block 4895 len 24576 162/300 Read block 4946 len 36864 163/300 Read block 2198 len 28672 164/300 Read block 2803 len 36864 165/300 Write block 3795 len 24576 data: 3 3 3 5 5 4 166/300 Flush 167/300 Write block 450 len 32768 data: 3 4 4 5 4 3 3 3 168/300 Read block 3665 len 40960 169/300 Flush 170/300 Write block 4887 len 40960 data: 8 6 3 3 3 3 3 3 3 3 171/300 Read block 4585 len 12288 172/300 Write block 237 len 4096 data: 4 173/300 Write block 3090 len 16384 data: 2 2 2 2 174/300 Read block 1615 len 32768 175/300 Read block 3692 len 40960 176/300 Read block 4734 len 20480 177/300 Write block 2030 len 28672 data: 5 6 6 6 4 4 5 178/300 Read block 3797 len 32768 179/300 Read block 1910 len 12288 180/300 Flush 181/300 Read block 894 len 8192 182/300 Read block 1466 len 20480 183/300 Read block 3226 len 16384 184/300 Read block 4086 len 24576 185/300 Write block 4503 len 36864 data: 6 6 5 5 4 4 4 3 3 186/300 Read block 3307 len 20480 187/300 Write block 3482 len 16384 data: 4 4 4 3 188/300 Write block 633 len 24576 data: 4 4 4 2 2 3 189/300 Read block 1859 len 16384 190/300 Flush 191/300 Write block 936 len 24576 data: 4 5 6 6 4 4 192/300 Write block 1018 len 28672 data: 2 2 2 5 5 5 4 193/300 Write block 3194 len 12288 data: 4 4 3 194/300 Flush 195/300 Read block 2632 len 28672 196/300 Write block 3077 len 40960 data: 8 6 5 5 5 6 6 5 4 6 197/300 Write block 2468 len 32768 data: 3 3 2 3 3 3 3 4 198/300 Write block 3906 len 32768 data: 4 4 5 4 5 4 4 4 199/300 Read block 4872 len 4096 200/300 Read block 3406 len 36864 201/300 Write block 2476 len 20480 data: 4 4 3 4 4 202/300 Read block 1777 len 20480 203/300 Read block 1304 len 16384 204/300 Read block 2558 len 8192 205/300 Write block 1750 len 20480 data: 2 3 3 3 4 206/300 Read block 2089 len 4096 207/300 Write block 3873 len 8192 data: 3 3 208/300 Read block 4762 len 32768 209/300 Read block 1056 len 20480 210/300 Read block 3538 len 28672 211/300 Write block 4619 len 36864 data: 3 2 2 2 2 2 3 6 5 212/300 Write block 204 len 24576 data: 3 4 4 5 6 7 213/300 Write block 3615 len 28672 data: 5 5 4 4 4 5 5 214/300 Write block 1136 len 8192 data: 7 7 215/300 Read block 4887 len 12288 216/300 Flush 217/300 Read block 935 len 8192 218/300 Read block 1648 len 4096 219/300 Read block 3463 len 4096 220/300 Write block 860 len 32768 data: 4 4 4 3 5 4 6 6 221/300 Write block 4637 len 24576 data: 6 4 5 5 4 3 222/300 Read block 4502 len 28672 223/300 Write block 906 len 16384 data: 2 2 2 2 224/300 Read block 4136 len 20480 225/300 Flush 226/300 Read block 3463 len 8192 227/300 Flush 228/300 Read block 2441 len 40960 229/300 Write block 3546 len 32768 data: 3 3 3 4 4 4 3 3 230/300 Read block 4335 len 28672 231/300 Write block 2702 len 40960 data: 2 3 3 5 5 3 4 4 5 4 232/300 Write block 2762 len 24576 data: 4 4 6 7 6 6 233/300 Flush 234/300 Write block 2147 len 32768 data: 4 4 3 3 3 4 4 5 235/300 Read block 1865 len 32768 236/300 Read block 2207 len 4096 237/300 Write block 4248 len 40960 data: 7 7 6 5 5 5 5 5 7 7 238/300 Read block 2650 len 36864 239/300 Write block 4434 len 28672 data: 6 4 4 5 5 4 3 240/300 Read block 317 len 24576 241/300 Read block 3758 len 8192 242/300 Read block 3512 len 12288 243/300 Write block 1188 len 24576 data: 5 5 4 3 3 3 244/300 Read block 4029 len 32768 245/300 Flush 246/300 Flush 247/300 Flush 248/300 Write block 535 len 32768 data: 7 7 6 5 5 4 4 4 249/300 Read block 237 len 28672 250/300 Write block 2235 len 4096 data: 5 251/300 Read block 4718 len 12288 252/300 Write block 4253 len 28672 data: 6 6 6 8 8 7 7 253/300 Read block 2529 len 40960 254/300 Read block 4342 len 40960 255/300 Read block 4486 len 28672 256/300 Write block 3299 len 28672 data: 2 3 3 3 3 3 3 257/300 Read block 4674 len 36864 258/300 Write block 3027 len 24576 data: 5 4 4 3 4 5 259/300 Read block 3483 len 36864 260/300 Read block 4001 len 8192 261/300 Write block 2104 len 28672 data: 3 3 2 2 2 2 3 262/300 Write block 4447 len 40960 data: 7 7 4 4 3 3 3 3 3 3 263/300 Read block 3864 len 40960 264/300 Write block 764 len 16384 data: 2 2 2 2 265/300 Read block 1084 len 8192 266/300 Write block 1217 len 28672 data: 3 4 2 2 3 3 4 267/300 Write block 3569 len 40960 data: 2 2 2 2 2 2 3 3 3 3 268/300 Flush 269/300 Read block 157 len 16384 270/300 Read block 3248 len 24576 271/300 Write block 172 len 4096 data: 4 272/300 Write block 1796 len 40960 data: 3 2 2 2 2 2 2 2 2 3 273/300 Write block 4818 len 8192 data: 3 3 274/300 Read block 2596 len 4096 275/300 Flush 276/300 Read block 3111 len 12288 277/300 Read block 4461 len 24576 278/300 Read block 3883 len 8192 279/300 Flush 280/300 Write block 4823 len 12288 data: 7 6 3 281/300 Read block 4564 len 28672 282/300 Read block 991 len 28672 283/300 Flush 284/300 Write block 3261 len 16384 data: 6 6 6 6 285/300 Write block 3708 len 28672 data: 3 4 3 3 3 3 4 286/300 Read block 1031 len 40960 287/300 Read block 1512 len 4096 288/300 Flush 289/300 Read block 3614 len 12288 290/300 Read block 3362 len 20480 291/300 Flush 292/300 Write block 2558 len 12288 data: 3 4 3 293/300 Flush 294/300 Read block 2704 len 40960 295/300 Flush 296/300 Read block 1174 len 32768 297/300 Write block 42 len 24576 data: 5 4 3 3 3 3 298/300 Read block 1316 len 28672 299/300 Read block 4069 len 40960 300/300 Read block 3608 len 4096 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:33:56.304 INFO request completed, latency_us: 288, response_code: 204, uri: /start/cid/0, method: POST, req_id: 975bfb08-5c33-45ce-af68-e0f52c6bd6c0, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 4835 Acked 5834 Read 1 New Done Done false 4836 Acked 5835 Read 4 New Done Done false 4837 Acked 5836 Write 2 New Done Done false 4838 Acked 5837 Read 9 New Done Done false 4839 Acked 5838 Read 7 New Done Done false 4840 Acked 5839 Read 5 New Done Done false 4841 Acked 5840 Write 7 New Done Done false 4842 Acked 5841 Read 3 New Done Done false 4843 Acked 5842 Read 2 New Done Done false 4844 Acked 5843 Read 4 New Done Done false 4845 Acked 5844 Write 8 New Done Done false 4846 Acked 5845 Read 1 New Done Done false 4847 Acked 5846 Read 6 New Done Done false 4848 Acked 5847 Write 7 New Done Done false 4849 Acked 5848 Flush 0 New Done Done false 4850 Acked 5849 Write 4 New Done Done false 4851 Acked 5850 Read 6 New Done Done false 4852 Acked 5851 Flush 0 New Done Done false 4853 Acked 5852 Read 9 New Done Done false 4854 Acked 5853 Write 3 New Done Done false 4855 Acked 5854 Read 3 New Done Done false 4856 Acked 5855 Flush 0 New Done Done false 4857 Acked 5856 Read 7 New Done Done false 4858 Acked 5857 Write 6 New Done Done false 4859 Acked 5858 Read 10 New Done Done false 4860 Acked 5859 Read 9 New Done Done false 4861 Acked 5860 Write 2 New Done Done false 4862 Acked 5861 Read 4 New Done Done false 4863 Acked 5862 Flush 0 New Done Done false 4864 Acked 5863 Read 6 New Done Done false 4865 Acked 5864 Write 4 New Done Done false 4866 Acked 5865 Read 5 New Done Done false 4867 Acked 5866 Read 9 New Done Done false 4868 Acked 5867 Flush 0 New Done Done false 4869 Acked 5868 Write 1 New Done Done false 4870 Acked 5869 Write 8 New Done Done false 4871 Acked 5870 Read 3 New Done Done false 4872 Acked 5871 Write 10 New Done Done false 4873 Acked 5872 Write 10 New Done Done false 4874 Acked 5873 Read 9 New Done Done false 4875 Acked 5874 Write 3 New Done Done false 4876 Acked 5875 Flush 0 New Done Done false 4877 Acked 5876 Write 7 New Done Done false 4878 Acked 5877 Write 2 New Done Done false 4879 Acked 5878 Flush 0 New Done Done false 4880 Acked 5879 Read 10 New Done Done false 4881 Acked 5880 Write 7 New Done Done false 4882 Acked 5881 Write 6 New Done Done false 4883 Acked 5882 Read 6 New Done Done false 4884 Acked 5883 Read 2 New Done Done false 4885 Acked 5884 Write 3 New Done Done false 4886 Acked 5885 Write 2 New Done Done false 4887 Acked 5886 Write 4 New Done Done false 4888 Acked 5887 Read 5 New Done Done false 4889 Acked 5888 Write 4 New Done Done false 4890 Acked 5889 Read 10 New Done Done false 4891 Acked 5890 Write 9 New Done Done false 4892 Acked 5891 Write 4 New Done Done false 4893 Acked 5892 Write 1 New Done Done false 4894 Acked 5893 Read 6 New Done Done false 4895 Acked 5894 Read 4 New Done Done false 4896 Acked 5895 Read 6 New Done Done false 4897 Acked 5896 Read 6 New Done Done false 4898 Acked 5897 Read 10 New Done Done false 4899 Acked 5898 Read 9 New Done Done false 4900 Acked 5899 Write 8 New Done Done false 4901 Acked 5900 Write 7 New Done Done false 4902 Acked 5901 Flush 0 New Done Done false 4903 Acked 5902 Write 1 New Done Done false 4904 Acked 5903 Read 6 New Done Done false 4905 Acked 5904 Read 2 New Done Done false 4906 Acked 5905 Write 1 New Done Done false 4907 Acked 5906 Read 7 New Done Done false 4908 Acked 5907 Flush 0 New Done Done false 4909 Acked 5908 Read 4 New Done Done false 4910 Acked 5909 Write 4 New Done Done false 4911 Acked 5910 Read 6 New Done Done false 4912 Acked 5911 Read 6 New Done Done false 4913 Acked 5912 Write 2 New Done Done false 4914 Acked 5913 Read 1 New Done Done false 4915 Acked 5914 Write 4 New Done Done false 4916 Acked 5915 Write 4 New Done Done false 4917 Acked 5916 Read 6 New Done Done false 4918 Acked 5917 Write 9 New Done Done false 4919 Acked 5918 Read 3 New Done Done false 4920 Acked 5919 Write 4 New Done Done false 4921 Acked 5920 Write 4 New Done Done false 4922 Acked 5921 Read 10 New Done Done false 4923 Acked 5922 Read 2 New Done Done false 4924 Acked 5923 Read 7 New Done Done false 4925 Acked 5924 Read 7 New Done Done false 4926 Acked 5925 Read 6 New Done Done false 4927 Acked 5926 Write 9 New Done Done false 4928 Acked 5927 Write 7 New Done Done false 4929 Acked 5928 Write 7 New Done Done false 4930 Acked 5929 Read 10 New Done Done false 4931 Acked 5930 Write 5 New Done Done false 4932 Acked 5931 Read 5 New Done Done false 4933 Acked 5932 Write 4 New Done Done false 4934 Acked 5933 Read 6 New Done Done false 4935 Acked 5934 Write 6 New Done Done false 4936 Acked 5935 Write 5 New Done Done false 4937 Acked 5936 Read 3 New Done Done false 4938 Acked 5937 Read 3 New Done Done false 4939 Acked 5938 Read 8 New Done Done false 4940 Acked 5939 Write 1 New Done Done false 4941 Acked 5940 Write 5 New Done Done false 4942 Acked 5941 Flush 0 New Done Done false 4943 Acked 5942 Read 3 New Done Done false 4944 Acked 5943 Read 3 New Done Done false 4945 Acked 5944 Flush 0 New Done Done false 4946 Acked 5945 Write 1 New Done Done false 4947 Acked 5946 Read 5 New Done Done false 4948 Acked 5947 Read 4 New Done Done false 4949 Acked 5948 Write 8 New Done Done false 4950 Acked 5949 Flush 0 New Done Done false 4951 Acked 5950 Read 9 New Done Done false 4952 Acked 5951 Write 4 New Done Done false 4953 Acked 5952 Flush 0 New Done Done false 4954 Acked 5953 Flush 0 New Done Done false 4955 Acked 5954 Read 4 New Done Done false 4956 Acked 5955 Flush 0 New Done Done false 4957 Acked 5956 Write 1 New Done Done false 4958 Acked 5957 Read 7 New Done Done false 4959 Acked 5958 Read 6 New Done Done false 4960 Acked 5959 Read 3 New Done Done false 4961 Acked 5960 Read 4 New Done Done false 4962 Acked 5961 Read 2 New Done Done false 4963 Acked 5962 Read 1 New Done Done false 4964 Acked 5963 Read 9 New Done Done false 4965 Acked 5964 Read 6 New Done Done false 4966 Acked 5965 Read 7 New Done Done false 4967 Acked 5966 Read 10 New Done Done false 4968 Acked 5967 Read 2 New Done Done false 4969 Acked 5968 Read 8 New Done Done false 4970 Acked 5969 Read 4 New Done Done false 4971 Acked 5970 Write 3 New Done Done false 4972 Acked 5971 Read 4 New Done Done false 4973 Acked 5972 Write 3 New Done Done false 4974 Acked 5973 Read 5 New Done Done false 4975 Acked 5974 Read 2 New Done Done false 4976 Acked 5975 Read 6 New Done Done false 4977 Acked 5976 Read 4 New Done Done false 4978 Acked 5977 Read 1 New Done Done false 4979 Acked 5978 Read 2 New Done Done false 4980 Acked 5979 Write 8 New Done Done false 4981 Acked 5980 Write 5 New Done Done false 4982 Acked 5981 Write 9 New Done Done false 4983 Acked 5982 Read 6 New Done Done false 4984 Acked 5983 Read 9 New Done Done false 4985 Acked 5984 Write 9 New Done Done false 4986 Acked 5985 Write 9 New Done Done false 4987 Acked 5986 Read 6 New Done Done false 4988 Acked 5987 Write 8 New Done Done false 4989 Acked 5988 Read 8 New Done Done false 4990 Acked 5989 Read 6 New Done Done false 4991 Acked 5990 Read 8 New Done Done false 4992 Acked 5991 Write 10 New Done Done false 4993 Acked 5992 Write 5 New Done Done false 4994 Acked 5993 Read 5 New Done Done false 4995 Acked 5994 Read 6 New Done Done false 4996 Acked 5995 Read 9 New Done Done false 4997 Acked 5996 Read 7 New Done Done false 4998 Acked 5997 Read 9 New Done Done false 4999 Acked 5998 Write 6 New Done Done false 5000 Acked 5999 Flush 0 New Done Done false 5001 Acked 6000 Write 8 New Done Done false 5002 Acked 6001 Read 10 New Done Done false 5003 Acked 6002 Flush 0 New Done Done false 5004 Acked 6003 Write 10 New Done Done false 5005 Acked 6004 Read 3 New Done Done false 5006 Acked 6005 Write 1 New Done Done false 5007 Acked 6006 Write 4 New Done Done false 5008 Acked 6007 Read 8 New Done Done false 5009 Acked 6008 Read 10 New Done Done false 5010 Acked 6009 Read 5 New Done Done false 5011 Acked 6010 Write 7 New Done Done false 5012 Acked 6011 Read 8 New Done Done false 5013 Acked 6012 Read 3 New Done Done false 5014 Acked 6013 Flush 0 New Done Done false 5015 Acked 6014 Read 2 New Done Done false 5016 Acked 6015 Read 5 New Done Done false 5017 Acked 6016 Read 4 New Done Done false 5018 Acked 6017 Read 6 New Done Done false 5019 Acked 6018 Write 9 New Done Done false 5020 Acked 6019 Read 5 New Done Done false 5021 Acked 6020 Write 4 New Done Done false 5022 Acked 6021 Write 6 New Done Done false 5023 Acked 6022 Read 4 New Done Done false 5024 Acked 6023 Flush 0 New Done Done false 5025 Acked 6024 Write 6 New Done Done false 5026 Acked 6025 Write 7 New Done Done false 5027 Acked 6026 Write 3 New Done Done false 5028 Acked 6027 Flush 0 New Done Done false 5029 Acked 6028 Read 7 New Done Done false 5030 Acked 6029 Write 10 New Done Done false 5031 Acked 6030 Write 8 New Done Done false 5032 Acked 6031 Write 8 New Done Done false 5033 Acked 6032 Read 1 New Done Done false 5034 Acked 6033 Read 9 New Done Done false 5035 Acked 6034 Write 5 New Done Done false 5036 Acked 6035 Read 5 New Done Done false 5037 Acked 6036 Read 4 New Done Done false 5038 Acked 6037 Read 2 New Done Done false 5039 Acked 6038 Write 5 New Done Done false 5040 Acked 6039 Read 1 New Done Done false 5041 Acked 6040 Write 2 New Done Done false 5042 Acked 6041 Read 8 New Done Done false 5043 Acked 6042 Read 5 New Done Done false 5044 Acked 6043 Read 7 New Done Done false 5045 Acked 6044 Write 9 New Done Done false 5046 Acked 6045 Write 6 New Done Done false 5047 Acked 6046 Write 7 New Done Done false 5048 Acked 6047 Write 2 New Done Done false 5049 Acked 6048 Read 3 New Done Done false 5050 Acked 6049 Flush 0 New Done Done false 5051 Acked 6050 Read 2 New Done Done false 5052 Acked 6051 Read 1 New Done Done false 5053 Acked 6052 Read 1 New Done Done false 5054 Acked 6053 Write 8 New Done Done false 5055 Acked 6054 Write 6 New Done Done false 5056 Acked 6055 Read 7 New Done Done false 5057 Acked 6056 Write 4 New Done Done false 5058 Acked 6057 Read 5 New Done Done false 5059 Acked 6058 Flush 0 New Done Done false 5060 Acked 6059 Read 2 New Done Done false 5061 Acked 6060 Flush 0 New Done Done false 5062 Acked 6061 Read 10 New Done Done false 5063 Acked 6062 Write 8 New Done Done false 5064 Acked 6063 Read 7 New Done Done false 5065 Acked 6064 Write 10 New Done Done false 5066 Acked 6065 Write 6 New Done Done false 5067 Acked 6066 Flush 0 New Done Done false 5068 Acked 6067 Write 8 New Done Done false 5069 Acked 6068 Read 8 New Done Done false 5070 Acked 6069 Read 1 New Done Done false 5071 Acked 6070 Write 10 New Done Done false 5072 Acked 6071 Read 9 New Done Done false 5073 Acked 6072 Write 7 New Done Done false 5074 Acked 6073 Read 6 New Done Done false 5075 Acked 6074 Read 2 New Done Done false 5076 Acked 6075 Read 3 New Done Done false 5077 Acked 6076 Write 6 New Done Done false 5078 Acked 6077 Read 8 New Done Done false 5079 Acked 6078 Flush 0 New Done Done false 5080 Acked 6079 Flush 0 New Done Done false 5081 Acked 6080 Flush 0 New Done Done false 5082 Acked 6081 Write 8 New Done Done false 5083 Acked 6082 Read 7 New Done Done false 5084 Acked 6083 Write 1 New Done Done false 5085 Acked 6084 Read 3 New Done Done false 5086 Acked 6085 Write 7 New Done Done false 5087 Acked 6086 Read 10 New Done Done false 5088 Acked 6087 Read 10 New Done Done false 5089 Acked 6088 Read 7 New Done Done false 5090 Acked 6089 Write 7 New Done Done false 5091 Acked 6090 Read 9 New Done Done false 5092 Acked 6091 Flush 0 New Done Done false 5093 Acked 6092 Write 6 New Done Done false 5094 Acked 6093 Read 9 New Done Done false 5095 Acked 6094 Read 2 New Done Done false 5096 Acked 6095 Write 7 New Done Done false 5097 Acked 6096 Write 10 New Done Done false 5098 Acked 6097 Read 10 New Done Done false 5099 Acked 6098 Write 4 New Done Done false 5100 Acked 6099 Read 2 New Done Done false 5101 Acked 6100 Write 7 New Done Done false 5102 Acked 6101 Write 10 New Done Done false 5103 Acked 6102 Flush 0 New Done Done false 5104 Acked 6103 Read 4 New Done Done false 5105 Acked 6104 Read 6 New Done Done false 5106 Acked 6105 Write 1 New Done Done false 5107 Acked 6106 Write 10 New Done Done false 5108 Acked 6107 Write 2 New Done Done false 5109 Acked 6108 Read 1 New Done Done false 5110 Acked 6109 Flush 0 New Done Done false 5111 Acked 6110 Read 3 New Done Done false 5112 Acked 6111 Read 6 New Done Done false 5113 Acked 6112 Read 2 New Done Done false 5114 Acked 6113 Flush 0 New Done Done false 5115 Acked 6114 Write 3 New Done Done false 5116 Acked 6115 Read 7 New Done Done false 5117 Acked 6116 Read 7 New Done Done false 5118 Acked 6117 Flush 0 New Done Done false 5119 Acked 6118 Write 4 New Done Done false 5120 Acked 6119 Write 7 New Done Done false 5121 Acked 6120 Read 10 New Done Done false 5122 Acked 6121 Read 1 New Done Done false 5123 Acked 6122 Flush 0 New Done Done false 5124 Acked 6123 Read 3 New Done Done false 5125 Acked 6124 Read 5 New Done Done false 5126 Acked 6125 Flush 0 New Done Done false 5127 Acked 6126 Write 3 New Done Done false 5128 Acked 6127 Flush 0 New Done Done false 5129 Acked 6128 Read 10 New Done Done false 5130 Acked 6129 Flush 0 New Done Done false 5131 Acked 6130 Read 8 New Done Done false 5132 Acked 6131 Write 6 New Done Done false 5133 Acked 6132 Read 7 New Done Done false 5134 Acked 6133 Read 10 New Done Done false 5135 Acked 6134 Read 1 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5833 6129 6129 Downstairs last five completed: 5833 5832 5831 5830 5829 Upstairs last five completed: 5135 5134 5133 5132 5131 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:56.687182944Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:56.68725433Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:56.687294865Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:56.687320502Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:56.687346738Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:56.687367134Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:56.687389661Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:56.68762342Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:57.689002658Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:57.68907352Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:57.689101656Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:57.689128335Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:57.689148535Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:57.689183996Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:57.689453855Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:58.690203185Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:58.690281902Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:58.690310125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:58.690336349Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:58.690356568Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:58.690401696Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:58.690653862Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:59.691424142Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:59.691494635Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:59.691522471Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:33:59.691548912Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:59.691569274Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:59.691603574Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:33:59.691868637Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 4835 Acked 5834 Read 1 New Done Done true 4836 Acked 5835 Read 4 New Done Done true 4837 Acked 5836 Write 2 New Done Done true 4838 Acked 5837 Read 9 New Done Done true 4839 Acked 5838 Read 7 New Done Done true 4840 Acked 5839 Read 5 New Done Done true 4841 Acked 5840 Write 7 New Done Done true 4842 Acked 5841 Read 3 New Done Done true 4843 Acked 5842 Read 2 New Done Done true 4844 Acked 5843 Read 4 New Done Done true 4845 Acked 5844 Write 8 New Done Done true 4846 Acked 5845 Read 1 New Done Done true 4847 Acked 5846 Read 6 New Done Done true 4848 Acked 5847 Write 7 New Done Done true 4849 Acked 5848 Flush 0 New Done Done true 4850 Acked 5849 Write 4 New Done Done true 4851 Acked 5850 Read 6 New Done Done true 4852 Acked 5851 Flush 0 New Done Done true 4853 Acked 5852 Read 9 New Done Done true 4854 Acked 5853 Write 3 New Done Done true 4855 Acked 5854 Read 3 New Done Done true 4856 Acked 5855 Flush 0 New Done Done true 4857 Acked 5856 Read 7 New Done Done true 4858 Acked 5857 Write 6 New Done Done true 4859 Acked 5858 Read 10 New Done Done true 4860 Acked 5859 Read 9 New Done Done true 4861 Acked 5860 Write 2 New Done Done true 4862 Acked 5861 Read 4 New Done Done true 4863 Acked 5862 Flush 0 New Done Done true 4864 Acked 5863 Read 6 New Done Done true 4865 Acked 5864 Write 4 New Done Done true 4866 Acked 5865 Read 5 New Done Done true 4867 Acked 5866 Read 9 New Done Done true 4868 Acked 5867 Flush 0 New Done Done true 4869 Acked 5868 Write 1 New Done Done true 4870 Acked 5869 Write 8 New Done Done true 4871 Acked 5870 Read 3 New Done Done true 4872 Acked 5871 Write 10 New Done Done true 4873 Acked 5872 Write 10 New Done Done true 4874 Acked 5873 Read 9 New Done Done true 4875 Acked 5874 Write 3 New Done Done true 4876 Acked 5875 Flush 0 New Done Done true 4877 Acked 5876 Write 7 New Done Done true 4878 Acked 5877 Write 2 New Done Done true 4879 Acked 5878 Flush 0 New Done Done true 4880 Acked 5879 Read 10 New Done Done true 4881 Acked 5880 Write 7 New Done Done true 4882 Acked 5881 Write 6 New Done Done true 4883 Acked 5882 Read 6 New Done Done true 4884 Acked 5883 Read 2 New Done Done true 4885 Acked 5884 Write 3 New Done Done true 4886 Acked 5885 Write 2 New Done Done true 4887 Acked 5886 Write 4 New Done Done true 4888 Acked 5887 Read 5 New Done Done true 4889 Acked 5888 Write 4 New Done Done true 4890 Acked 5889 Read 10 New Done Done true 4891 Acked 5890 Write 9 New Done Done true 4892 Acked 5891 Write 4 New Done Done true 4893 Acked 5892 Write 1 New Done Done true 4894 Acked 5893 Read 6 New Done Done true 4895 Acked 5894 Read 4 New Done Done true 4896 Acked 5895 Read 6 New Done Done true 4897 Acked 5896 Read 6 New Done Done true 4898 Acked 5897 Read 10 New Done Done true 4899 Acked 5898 Read 9 New Done Done true 4900 Acked 5899 Write 8 New Done Done true 4901 Acked 5900 Write 7 New Done Done true 4902 Acked 5901 Flush 0 New Done Done true 4903 Acked 5902 Write 1 New Done Done true 4904 Acked 5903 Read 6 New Done Done true 4905 Acked 5904 Read 2 New Done Done true 4906 Acked 5905 Write 1 New Done Done true 4907 Acked 5906 Read 7 New Done Done true 4908 Acked 5907 Flush 0 New Done Done true 4909 Acked 5908 Read 4 New Done Done true 4910 Acked 5909 Write 4 New Done Done true 4911 Acked 5910 Read 6 New Done Done true 4912 Acked 5911 Read 6 New Done Done true 4913 Acked 5912 Write 2 New Done Done true 4914 Acked 5913 Read 1 New Done Done true 4915 Acked 5914 Write 4 New Done Done true 4916 Acked 5915 Write 4 New Done Done true 4917 Acked 5916 Read 6 New Done Done true 4918 Acked 5917 Write 9 New Done Done true 4919 Acked 5918 Read 3 New Done Done true 4920 Acked 5919 Write 4 New Done Done true 4921 Acked 5920 Write 4 New Done Done true 4922 Acked 5921 Read 10 New Done Done true 4923 Acked 5922 Read 2 New Done Done true 4924 Acked 5923 Read 7 New Done Done true 4925 Acked 5924 Read 7 New Done Done true 4926 Acked 5925 Read 6 New Done Done true 4927 Acked 5926 Write 9 New Done Done true 4928 Acked 5927 Write 7 New Done Done true 4929 Acked 5928 Write 7 New Done Done true 4930 Acked 5929 Read 10 New Done Done true 4931 Acked 5930 Write 5 New Done Done true 4932 Acked 5931 Read 5 New Done Done true 4933 Acked 5932 Write 4 New Done Done true 4934 Acked 5933 Read 6 New Done Done true 4935 Acked 5934 Write 6 New Done Done true 4936 Acked 5935 Write 5 New Done Done true 4937 Acked 5936 Read 3 New Done Done true 4938 Acked 5937 Read 3 New Done Done true 4939 Acked 5938 Read 8 New Done Done true 4940 Acked 5939 Write 1 New Done Done true 4941 Acked 5940 Write 5 New Done Done true 4942 Acked 5941 Flush 0 New Done Done true 4943 Acked 5942 Read 3 New Done Done true 4944 Acked 5943 Read 3 New Done Done true 4945 Acked 5944 Flush 0 New Done Done true 4946 Acked 5945 Write 1 New Done Done true 4947 Acked 5946 Read 5 New Done Done true 4948 Acked 5947 Read 4 New Done Done true 4949 Acked 5948 Write 8 New Done Done true 4950 Acked 5949 Flush 0 New Done Done true 4951 Acked 5950 Read 9 New Done Done true 4952 Acked 5951 Write 4 New Done Done true 4953 Acked 5952 Flush 0 New Done Done true 4954 Acked 5953 Flush 0 New Done Done true 4955 Acked 5954 Read 4 New Done Done true 4956 Acked 5955 Flush 0 New Done Done true 4957 Acked 5956 Write 1 New Done Done true 4958 Acked 5957 Read 7 New Done Done true 4959 Acked 5958 Read 6 New Done Done true 4960 Acked 5959 Read 3 New Done Done true 4961 Acked 5960 Read 4 New Done Done true 4962 Acked 5961 Read 2 New Done Done true 4963 Acked 5962 Read 1 New Done Done true 4964 Acked 5963 Read 9 New Done Done true 4965 Acked 5964 Read 6 New Done Done true 4966 Acked 5965 Read 7 New Done Done true 4967 Acked 5966 Read 10 New Done Done true 4968 Acked 5967 Read 2 New Done Done true 4969 Acked 5968 Read 8 New Done Done true 4970 Acked 5969 Read 4 New Done Done true 4971 Acked 5970 Write 3 New Done Done true 4972 Acked 5971 Read 4 New Done Done true 4973 Acked 5972 Write 3 New Done Done true 4974 Acked 5973 Read 5 New Done Done true 4975 Acked 5974 Read 2 New Done Done true 4976 Acked 5975 Read 6 New Done Done true 4977 Acked 5976 Read 4 New Done Done true 4978 Acked 5977 Read 1 New Done Done true 4979 Acked 5978 Read 2 New Done Done true 4980 Acked 5979 Write 8 New Done Done true 4981 Acked 5980 Write 5 New Done Done true 4982 Acked 5981 Write 9 New Done Done true 4983 Acked 5982 Read 6 New Done Done true 4984 Acked 5983 Read 9 New Done Done true 4985 Acked 5984 Write 9 New Done Done true 4986 Acked 5985 Write 9 New Done Done true 4987 Acked 5986 Read 6 New Done Done true 4988 Acked 5987 Write 8 New Done Done true 4989 Acked 5988 Read 8 New Done Done true 4990 Acked 5989 Read 6 New Done Done true 4991 Acked 5990 Read 8 New Done Done true 4992 Acked 5991 Write 10 New Done Done true 4993 Acked 5992 Write 5 New Done Done true 4994 Acked 5993 Read 5 New Done Done true 4995 Acked 5994 Read 6 New Done Done true 4996 Acked 5995 Read 9 New Done Done true 4997 Acked 5996 Read 7 New Done Done true 4998 Acked 5997 Read 9 New Done Done true 4999 Acked 5998 Write 6 New Done Done true 5000 Acked 5999 Flush 0 New Done Done true 5001 Acked 6000 Write 8 New Done Done true 5002 Acked 6001 Read 10 New Done Done true 5003 Acked 6002 Flush 0 New Done Done true 5004 Acked 6003 Write 10 New Done Done true 5005 Acked 6004 Read 3 New Done Done true 5006 Acked 6005 Write 1 New Done Done true 5007 Acked 6006 Write 4 New Done Done true 5008 Acked 6007 Read 8 New Done Done true 5009 Acked 6008 Read 10 New Done Done true 5010 Acked 6009 Read 5 New Done Done true 5011 Acked 6010 Write 7 New Done Done true 5012 Acked 6011 Read 8 New Done Done true 5013 Acked 6012 Read 3 New Done Done true 5014 Acked 6013 Flush 0 New Done Done true 5015 Acked 6014 Read 2 New Done Done true 5016 Acked 6015 Read 5 New Done Done true 5017 Acked 6016 Read 4 New Done Done true 5018 Acked 6017 Read 6 New Done Done true 5019 Acked 6018 Write 9 New Done Done true 5020 Acked 6019 Read 5 New Done Done true 5021 Acked 6020 Write 4 New Done Done true 5022 Acked 6021 Write 6 New Done Done true 5023 Acked 6022 Read 4 New Done Done true 5024 Acked 6023 Flush 0 New Done Done true 5025 Acked 6024 Write 6 New Done Done true 5026 Acked 6025 Write 7 New Done Done true 5027 Acked 6026 Write 3 New Done Done true 5028 Acked 6027 Flush 0 New Done Done true 5029 Acked 6028 Read 7 New Done Done true 5030 Acked 6029 Write 10 New Done Done true 5031 Acked 6030 Write 8 New Done Done true 5032 Acked 6031 Write 8 New Done Done true 5033 Acked 6032 Read 1 New Done Done true 5034 Acked 6033 Read 9 New Done Done true 5035 Acked 6034 Write 5 New Done Done true 5036 Acked 6035 Read 5 New Done Done true 5037 Acked 6036 Read 4 New Done Done true 5038 Acked 6037 Read 2 New Done Done true 5039 Acked 6038 Write 5 New Done Done true 5040 Acked 6039 Read 1 New Done Done true 5041 Acked 6040 Write 2 New Done Done true 5042 Acked 6041 Read 8 New Done Done true 5043 Acked 6042 Read 5 New Done Done true 5044 Acked 6043 Read 7 New Done Done true 5045 Acked 6044 Write 9 New Done Done true 5046 Acked 6045 Write 6 New Done Done true 5047 Acked 6046 Write 7 New Done Done true 5048 Acked 6047 Write 2 New Done Done true 5049 Acked 6048 Read 3 New Done Done true 5050 Acked 6049 Flush 0 New Done Done true 5051 Acked 6050 Read 2 New Done Done true 5052 Acked 6051 Read 1 New Done Done true 5053 Acked 6052 Read 1 New Done Done true 5054 Acked 6053 Write 8 New Done Done true 5055 Acked 6054 Write 6 New Done Done true 5056 Acked 6055 Read 7 New Done Done true 5057 Acked 6056 Write 4 New Done Done true 5058 Acked 6057 Read 5 New Done Done true 5059 Acked 6058 Flush 0 New Done Done true 5060 Acked 6059 Read 2 New Done Done true 5061 Acked 6060 Flush 0 New Done Done true 5062 Acked 6061 Read 10 New Done Done true 5063 Acked 6062 Write 8 New Done Done true 5064 Acked 6063 Read 7 New Done Done true 5065 Acked 6064 Write 10 New Done Done true 5066 Acked 6065 Write 6 New Done Done true 5067 Acked 6066 Flush 0 New Done Done true 5068 Acked 6067 Write 8 New Done Done true 5069 Acked 6068 Read 8 New Done Done true 5070 Acked 6069 Read 1 New Done Done true 5071 Acked 6070 Write 10 New Done Done true 5072 Acked 6071 Read 9 New Done Done true 5073 Acked 6072 Write 7 New Done Done true 5074 Acked 6073 Read 6 New Done Done true 5075 Acked 6074 Read 2 New Done Done true 5076 Acked 6075 Read 3 New Done Done true 5077 Acked 6076 Write 6 New Done Done true 5078 Acked 6077 Read 8 New Done Done true 5079 Acked 6078 Flush 0 New Done Done true 5080 Acked 6079 Flush 0 New Done Done true 5081 Acked 6080 Flush 0 New Done Done true 5082 Acked 6081 Write 8 New Done Done true 5083 Acked 6082 Read 7 New Done Done true 5084 Acked 6083 Write 1 New Done Done true 5085 Acked 6084 Read 3 New Done Done true 5086 Acked 6085 Write 7 New Done Done true 5087 Acked 6086 Read 10 New Done Done true 5088 Acked 6087 Read 10 New Done Done true 5089 Acked 6088 Read 7 New Done Done true 5090 Acked 6089 Write 7 New Done Done true 5091 Acked 6090 Read 9 New Done Done true 5092 Acked 6091 Flush 0 New Done Done true 5093 Acked 6092 Write 6 New Done Done true 5094 Acked 6093 Read 9 New Done Done true 5095 Acked 6094 Read 2 New Done Done true 5096 Acked 6095 Write 7 New Done Done true 5097 Acked 6096 Write 10 New Done Done true 5098 Acked 6097 Read 10 New Done Done true 5099 Acked 6098 Write 4 New Done Done true 5100 Acked 6099 Read 2 New Done Done true 5101 Acked 6100 Write 7 New Done Done true 5102 Acked 6101 Write 10 New Done Done true 5103 Acked 6102 Flush 0 New Done Done true 5104 Acked 6103 Read 4 New Done Done true 5105 Acked 6104 Read 6 New Done Done true 5106 Acked 6105 Write 1 New Done Done true 5107 Acked 6106 Write 10 New Done Done true 5108 Acked 6107 Write 2 New Done Done true 5109 Acked 6108 Read 1 New Done Done true 5110 Acked 6109 Flush 0 New Done Done true 5111 Acked 6110 Read 3 New Done Done true 5112 Acked 6111 Read 6 New Done Done true 5113 Acked 6112 Read 2 New Done Done true 5114 Acked 6113 Flush 0 New Done Done true 5115 Acked 6114 Write 3 New Done Done true 5116 Acked 6115 Read 7 New Done Done true 5117 Acked 6116 Read 7 New Done Done true 5118 Acked 6117 Flush 0 New Done Done true 5119 Acked 6118 Write 4 New Done Done true 5120 Acked 6119 Write 7 New Done Done true 5121 Acked 6120 Read 10 New Done Done true 5122 Acked 6121 Read 1 New Done Done true 5123 Acked 6122 Flush 0 New Done Done true 5124 Acked 6123 Read 3 New Done Done true 5125 Acked 6124 Read 5 New Done Done true 5126 Acked 6125 Flush 0 New Done Done true 5127 Acked 6126 Write 3 New Done Done true 5128 Acked 6127 Flush 0 New Done Done true 5129 Acked 6128 Read 10 New Done Done true 5130 Acked 6129 Flush 0 New Done Done true 5131 Acked 6130 Read 8 New Done Done true 5132 Acked 6131 Write 6 New Done Done true 5133 Acked 6132 Read 7 New Done Done true 5134 Acked 6133 Read 10 New Done Done true 5135 Acked 6134 Read 1 New Done Done true 5136 Acked 6135 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 5833 6135 6135 Downstairs last five completed: 5833 5832 5831 5830 5829 Upstairs last five completed: 5136 5135 5134 5133 5132 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:00.692817461Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:00.692886391Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:00.692914215Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:00.692940537Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:00.6929718Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:00.692995728Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:00.693252215Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:01.694952835Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:01.695020114Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:01.695047697Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:01.695071507Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:01.695097577Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:01.695118101Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:01.695148244Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:01.69540674Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1202) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.696527073Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.696631094Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.698675803Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.698707352Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.698730842Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 5833","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.699075677Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.69910561Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.699129141Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.699151577Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:02.699173765Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 6135 6134 6133 6132 6131 Upstairs last five completed: 5136 5135 5134 5133 5132 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(1) stop 1 Jan 02 22:34:04.310 INFO request completed, latency_us: 491, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 66289fe6-140e-4f6f-ba0b-943b98fc160e, remote_addr: 127.0.0.1:37434[8820] Got stop action so:false kr:false , local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:04.311996185Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:04.312114615Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:04.312158712Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:04.312181823Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:34:04.312 INFO request completed, latency_us: 241, response_code: 200, uri: /state/cid/1, method: GET, req_id{"msg":"[1] client re-new 0 jobs since flush 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:04.312275413Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} : b830b404-3837-4295-87dd-68453ad6c40b, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:04.312345496Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:05.313952285Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:05.314064323Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:05.314126415Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:05.314201011Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:05.314221678Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:05.314244198Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:05.314266426Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:06.315256924Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:06.315322399Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:06.315350802Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:06.31538526Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:06.315412803Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:06.315433409Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:06.315455716Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:06.315477868Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:07.316422983Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:07.316489499Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:07.316517324Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:07.31654367Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:07.31656429Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:07.316594818Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:07.316702504Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:34:08.314 INFO request completed, latency_us: 266, response_code: 200, uri: /state/cid/1, method: GET, req_id: a2f99a6e-db2d-4aa1-abcb-b564bc77b189, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 2139 len 20480 002/300 Read block 2874 len 8192 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:08.31800194Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:08.318072959Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:08.31810273Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:08.318130839Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:08.318168071Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 2 jobs since flush 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:08.318213347Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:08.318253143Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 2471 len 4096 004/300 Read block 1028 len 4096 005/300 Read block 289 len 40960 006/300 Read block 2627 len 16384 007/300 Read block 610 len 40960 008/300 Write block 4600 len 4096 data: 3 009/300 Read block 3032 len 8192 010/300 Read block 357 len 32768 011/300 Write block 3984 len 24576 data: 5 6 5 6 6 6 012/300 Read block 802 len 16384 013/300 Read block 3593 len 8192 014/300 Read block 1770 len 36864 015/300 Read block 4363 len 8192 016/300 Write block 2462 len 40960 data: 6 5 4 4 4 3 4 4 3 4 017/300 Write block 1014 len 28672 data: 5 3 3 3 3 3 3 018/300 Read block 186 len 4096 019/300 Flush 020/300 Write block 3711 len 4096 data: 4 021/300 Write block 428 len 36864 data: 4 5 5 6 5 4 4 4 7 022/300 Write block 381 len 40960 data: 4 4 3 3 2 2 2 3 3 2 023/300 Write block 264 len 28672 data: 4 4 4 6 7 6 6 024/300 Read block 4860 len 32768 025/300 Read block 1194 len 16384 026/300 Write block 4736 len 28672 data: 8 8 8 6 7 6 5 027/300 Flush 028/300 Write block 4836 len 8192 data: 3 3 029/300 Write block 4820 len 28672 data: 4 4 6 8 7 4 4 030/300 Write block 2076 len 12288 data: 2 3 3 031/300 Read block 2612 len 28672 032/300 Read block 3720 len 24576 033/300 Read block 2382 len 40960 034/300 Read block 1120 len 8192 035/300 Flush 036/300 Write block 520 len 12288 data: 2 2 3 037/300 Flush 038/300 Write block 4071 len 20480 data: 2 2 3 3 3 039/300 Write block 2093 len 24576 data: 2 2 2 3 3 4 040/300 Flush 041/300 Flush 042/300 Write block 3187 len 4096 data: 3 043/300 Write block 422 len 4096 data: 6 044/300 Write block 64 len 16384 data: 4 3 3 3 045/300 Read block 2345 len 4096 046/300 Write block 1911 len 32768 data: 3 4 3 3 3 3 3 4 047/300 Read block 1987 len 36864 048/300 Write block 1476 len 4096 data: 2 049/300 Write block 1866 len 40960 data: 4 4 3 3 4 4 3 3 4 4 050/300 Write block 2250 len 40960 data: 2 2 2 2 2 4 6 5 5 7 051/300 Flush 052/300 Write block 3212 len 16384 data: 2 2 2 2 053/300 Write block 822 len 12288 data: 3 3 3 054/300 Read block 838 len 4096 055/300 Read block 12 len 12288 056/300 Write block 1949 len 12288 data: 5 3 2 057/300 Read block 4891 len 28672 058/300 Write block 3657 len 40960 data: 4 4 4 4 4 3 4 3 3 4 059/300 Write block 1843 len 36864 data: 2 2 2 2 2 2 2 2 2 060/300 Write block 3242 len 16384 data: 4 5 4 3 061/300 Flush 062/300 Write block 4620 len 16384 data: 3 3 3 3 063/300 Write block 3513 len 4096 data: 4 064/300 Read block 2058 len 40960 065/300 Write block 3243 len 28672 data: 6 5 4 5 5 5 5 066/300 Read block 4080 len 4096 067/300 Write block 390 len 28672 data: 3 4 5 6 7 7 4 068/300 Write block 118 len 32768 data: 3 3 3 4 4 3 3 4 069/300 Write block 1034 len 8192 data: 3 4 070/300 Write block 1229 len 28672 data: 6 5 5 5 4 3 3 071/300 Write block 3142 len 16384 data: 4 4 3 2 072/300 Read block 4413 len 8192 073/300 Write block 225 len 4096 data: 4 074/300 Write block 484 len 24576 data: 3 3 3 3 3 3 075/300 Write block 1343 len 16384 data: 4 5 5 6 076/300 Read block 3461 len 40960 077/300 Read block 3819 len 4096 078/300 Write block 4040 len 24576 data: 4 4 5 6 5 6 079/300 Write block 1610 len 16384 data: 3 3 3 3 080/300 Read block 1929 len 36864 081/300 Read block 2206 len 4096 082/300 Write block 524 len 12288 data: 4 6 6 083/300 Read block 1789 len 4096 084/300 Read block 2830 len 4096 085/300 Write block 2344 len 4096 data: 4 086/300 Read block 1556 len 8192 087/300 Read block 1922 len 36864 088/300 Read block 2231 len 20480 089/300 Write block 475 len 16384 data: 6 6 6 6 090/300 Read block 733 len 4096 091/300 Read block 2246 len 28672 092/300 Read block 1359 len 32768 093/300 Read block 4354 len 16384 094/300 Read block 2843 len 28672 095/300 Flush 096/300 Read block 643 len 28672 097/300 Read block 1616 len 28672 098/300 Write block 1463 len 36864 data: 5 4 4 3 3 3 3 2 2 099/300 Write block 1176 len 28672 data: 4 4 3 3 3 4 3 100/300 Write block 4095 len 4096 data: 4 101/300 Read block 3797 len 4096 102/300 Write block 518 len 20480 data: 4 3 3 3 4 103/300 Write block 2685 len 36864 data: 5 4 5 7 6 7 7 7 5 104/300 Write block 2822 len 36864 data: 8 9 8 9 9 8 7 5 5 105/300 Read block 1701 len 24576 106/300 Write block 139 len 24576 data: 3 2 2 2 2 4 107/300 Write block 4134 len 40960 data: 5 4 2 3 3 3 3 4 3 6 108/300 Write block 2818 len 24576 data: 4 5 6 8 9 10 109/300 Flush 110/300 Read block 1961 len 20480 111/300 Flush 112/300 Read block 4895 len 16384 113/300 Write block 4258 len 20480 data: 8 8 8 7 6 114/300 Read block 1145 len 8192 115/300 Write block 1781 len 36864 data: 5 2 2 3 3 3 3 3 4 116/300 Write block 1668 len 20480 data: 4 5 4 4 3 117/300 Write block 2934 len 12288 data: 6 5 6 118/300 Read block 988 len 8192 119/300 Write block 670 len 40960 data: 6 5 3 2 2 3 3 2 2 2 120/300 Read block 1384 len 20480 121/300 Write block 4307 len 16384 data: 5 4 4 4 122/300 Write block 1285 len 20480 data: 4 6 5 7 5 123/300 Read block 1239 len 32768 124/300 Read block 675 len 4096 125/300 Read block 3682 len 40960 126/300 Flush 127/300 Read block 3287 len 4096 128/300 Write block 2831 len 40960 data: 3 2 2 2 3 3 4 5 5 5 129/300 Write block 1966 len 28672 data: 4 3 3 4 4 4 4 130/300 Read block 1178 len 20480 131/300 Flush 132/300 Write block 1072 len 28672 data: 7 7 7 6 5 7 6 133/300 Read block 3681 len 24576 134/300 Read block 4709 len 16384 135/300 Write block 656 len 40960 data: 6 6 7 8 9 8 8 10 10 10 136/300 Write block 4642 len 28672 data: 4 4 4 4 4 4 5 137/300 Write block 3348 len 8192 data: 7 6 138/300 Write block 3059 len 36864 data: 6 6 4 6 5 5 3 4 4 139/300 Write block 2610 len 8192 data: 6 3 140/300 Read block 1535 len 16384 141/300 Write block 2198 len 20480 data: 5 4 4 4 2 142/300 Read block 4419 len 8192 143/300 Write block 1757 len 12288 data: 7 7 9 144/300 Write block 3481 len 28672 data: 4 5 5 5 4 3 3 145/300 Write block 3234 len 36864 data: 4 5 6 7 5 5 6 6 5 146/300 Read block 2769 len 20480 147/300 Read block 4429 len 28672 148/300 Write block 691 len 24576 data: 6 3 3 3 3 4 149/300 Write block 1476 len 40960 data: 3 2 2 3 2 2 2 2 3 3 150/300 Flush 151/300 Write block 1137 len 36864 data: 8 6 4 3 5 4 3 2 3 152/300 Flush 153/300 Read block 4958 len 36864 154/300 Read block 2651 len 20480 155/300 Write block 3839 len 40960 data: 4 4 5 4 4 4 3 3 3 3 156/300 Write block 4007 len 20480 data: 5 6 8 9 7 157/300 Write block 544 len 28672 data: 2 3 4 4 4 4 4 158/300 Write block 1460 len 8192 data: 3 4 159/300 Write block 4562 len 36864 data: 4 5 5 5 5 5 5 5 6 160/300 Read block 3672 len 28672 161/300 Write block 4074 len 20480 data: 4 4 4 3 4 162/300 Read block 4406 len 32768 163/300 Read block 989 len 12288 164/300 Write block 3330 len 12288 data: 3 3 6 165/300 Read block 1339 len 12288 166/300 Write block 1683 len 36864 data: 6 5 6 5 6 6 4 4 4 167/300 Write block 2682 len 8192 data: 2 2 168/300 Write block 3304 len 16384 data: 4 4 4 3 169/300 Read block 3927 len 36864 170/300 Read block 1946 len 28672 171/300 Read block 3961 len 32768 172/300 Flush 173/300 Write block 1529 len 20480 data: 3 3 3 3 4 174/300 Read block 1652 len 12288 175/300 Read block 877 len 36864 176/300 Read block 239 len 8192 177/300 Flush 178/300 Read block 3645 len 36864 179/300 Read block 1800 len 8192 180/300 Write block 2116 len 32768 data: 3 3 3 4 3 4 5 5 181/300 Write block 1629 len 28672 data: 5 5 6 4 4 4 5 182/300 Write block 4119 len 12288 data: 4 4 4 183/300 Write block 2283 len 20480 data: 5 4 3 3 3 184/300 Read block 344 len 12288 185/300 Write block 3823 len 20480 data: 3 2 2 2 3 186/300 Read block 2615 len 20480 187/300 Write block 2643 len 8192 data: 5 5 188/300 Read block 2381 len 4096 189/300 Flush 190/300 Read block 3295 len 4096 191/300 Read block 889 len 4096 192/300 Write block 4801 len 40960 data: 5 5 6 6 7 6 5 4 3 3 193/300 Write block 3817 len 20480 data: 2 2 2 2 3 194/300 Write block 1219 len 28672 data: 3 3 4 4 5 5 5 195/300 Write block 199 len 24576 data: 2 3 3 3 3 4 196/300 Write block 3937 len 12288 data: 2 2 3 197/300 Read block 4608 len 36864 198/300 Read block 438 len 32768 199/300 Read block 2290 len 24576 200/300 Write block 2640 len 20480 data: 6 5 5 6 6 201/300 Read block 2086 len 8192 202/300 Write block 3160 len 20480 data: 2 2 2 2 3 203/300 Write block 1162 len 8192 data: 5 5 204/300 Read block 3597 len 12288 205/300 Read block 1692 len 28672 206/300 Read block 4988 len 36864 207/300 Write block 2335 len 40960 data: 5 5 6 5 4 5 3 4 4 5 208/300 Read block 1462 len 4096 209/300 Read block 1150 len 36864 210/300 Write block 2903 len 32768 data: 4 3 3 5 4 4 3 3 211/300 Read block 2511 len 36864 212/300 Read block 2145 len 24576 213/300 Read block 2847 len 20480 214/300 Flush 215/300 Read block 1644 len 12288 216/300 Read block 4425 len 36864 217/300 Read block 2979 len 32768 218/300 Write block 3513 len 24576 data: 5 3 3 3 3 3 219/300 Write block 4220 len 16384 data: 6 5 5 5 220/300 Read block 1704 len 20480 221/300 Write block 492 len 28672 data: 4 4 4 4 2 2 3 222/300 Read block 4896 len 20480 223/300 Read block 2201 len 36864 224/300 Read block 3269 len 4096 225/300 Read block 720 len 16384 226/300 Read block 866 len 24576 227/300 Write block 1986 len 16384 data: 5 4 3 3 228/300 Read block 377 len 36864 229/300 Read block 464 len 8192 230/300 Write block 1533 len 20480 data: 5 4 4 4 3 231/300 Write block 3757 len 36864 data: 3 4 3 3 3 3 5 6 8 232/300 Flush 233/300 Write block 3147 len 24576 data: 4 4 5 5 4 5 234/300 Read block 2738 len 8192 235/300 Write block 2479 len 24576 data: 5 5 3 3 4 3 236/300 Write block 779 len 20480 data: 3 3 3 3 3 237/300 Write block 4695 len 12288 data: 6 6 6 238/300 Read block 4950 len 16384 239/300 Write block 2747 len 36864 data: 4 4 3 3 3 4 5 5 5 240/300 Write block 4336 len 16384 data: 4 5 4 5 241/300 Read block 2154 len 32768 242/300 Read block 3683 len 32768 243/300 Read block 2459 len 28672 244/300 Write block 4046 len 40960 data: 6 7 6 6 5 4 4 3 2 2 245/300 Read block 2843 len 8192 246/300 Flush 247/300 Read block 4343 len 16384 248/300 Write block 4345 len 4096 data: 6 249/300 Read block 2113 len 36864 250/300 Read block 2058 len 12288 251/300 Write block 3788 len 20480 data: 5 5 4 5 5 252/300 Read block 4794 len 16384 253/300 Read block 846 len 24576 254/300 Read block 1854 len 20480 255/300 Read block 2340 len 20480 256/300 Read block 1809 len 24576 257/300 Write block 3314 len 32768 data: 2 2 3 3 4 5 5 4 258/300 Read block 4919 len 36864 259/300 Read block 4868 len 8192 260/300 Write block 3717 len 12288 data: 4 3 3 261/300 Read block 3466 len 36864 262/300 Write block 1349 len 24576 data: 7 8 6 6 7 6 263/300 Write block 1415 len 40960 data: 4 5 4 4 5 5 6 6 5 5 264/300 Write block 631 len 8192 data: 3 3 265/300 Write block 2660 len 12288 data: 3 2 3 266/300 Read block 3364 len 4096 267/300 Read block 2023 len 16384 268/300 Read block 839 len 36864 269/300 Write block 2463 len 24576 data: 6 5 5 5 4 5 270/300 Read block 3095 len 12288 271/300 Write block 3212 len 16384 data: 3 3 3 3 272/300 Read block 4703 len 4096 273/300 Read block 837 len 4096 274/300 Write block 469 len 12288 data: 4 5 4 275/300 Read block 2884 len 24576 276/300 Write block 3680 len 24576 data: 4 4 3 2 2 2 277/300 Read block 985 len 24576 278/300 Write block 4743 len 36864 data: 4 3 3 3 4 4 4 4 3 279/300 Write block 2540 len 4096 data: 3 280/300 Write block 4859 len 12288 data: 2 2 3 281/300 Read block 3740 len 4096 282/300 Read block 3978 len 32768 283/300 Flush 284/300 Read block 2205 len 4096 285/300 Read block 160 len 32768 286/300 Write block 1978 len 24576 data: 4 4 3 3 4 4 287/300 Write block 630 len 4096 data: 3 288/300 Write block 1191 len 8192 data: 4 4 289/300 Read block 3964 len 20480 290/300 Write block 873 len 4096 data: 3 291/300 Read block 4675 len 28672 292/300 Write block 3996 len 4096 data: 8 293/300 Read block 2912 len 8192 294/300 Read block 3461 len 40960 295/300 Write block 1089 len 40960 data: 3 3 3 3 3 4 4 4 3 4 296/300 Read block 1909 len 28672 297/300 Write block 3250 len 12288 data: 4 4 5 298/300 Read block 1252 len 8192 299/300 Write block 1782 len 4096 data: 3 300/300 Write block 2499 len 40960 data: 2 2 2 4 5 5 6 7 7 8 Main task has work to do, go find it got dsc Start(1) start 1 Jan 02 22:34:08.869 INFO request completed, latency_us: 296, response_code: 204, uri: /start/cid/1, method: POST, req_id: b675a10e-fcf4-429f-aaa3-a0f94a772544, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 5137 Acked 6136 Read 5 Done New Done true 5138 Acked 6137 Read 2 Done New Done true 5139 Acked 6138 Read 1 Done New Done false 5140 Acked 6139 Read 1 Done New Done false 5141 Acked 6140 Read 10 Done New Done false 5142 Acked 6141 Read 4 Done New Done false 5143 Acked 6142 Read 10 Done New Done false 5144 Acked 6143 Write 1 Done New Done false 5145 Acked 6144 Read 2 Done New Done false 5146 Acked 6145 Read 8 Done New Done false 5147 Acked 6146 Write 6 Done New Done false 5148 Acked 6147 Read 4 Done New Done false 5149 Acked 6148 Read 2 Done New Done false 5150 Acked 6149 Read 9 Done New Done false 5151 Acked 6150 Read 2 Done New Done false 5152 Acked 6151 Write 10 Done New Done false 5153 Acked 6152 Write 7 Done New Done false 5154 Acked 6153 Read 1 Done New Done false 5155 Acked 6154 Flush 0 Done New Done false 5156 Acked 6155 Write 1 Done New Done false 5157 Acked 6156 Write 9 Done New Done false 5158 Acked 6157 Write 10 Done New Done false 5159 Acked 6158 Write 7 Done New Done false 5160 Acked 6159 Read 8 Done New Done false 5161 Acked 6160 Read 4 Done New Done false 5162 Acked 6161 Write 7 Done New Done false 5163 Acked 6162 Flush 0 Done New Done false 5164 Acked 6163 Write 2 Done New Done false 5165 Acked 6164 Write 7 Done New Done false 5166 Acked 6165 Write 3 Done New Done false 5167 Acked 6166 Read 7 Done New Done false 5168 Acked 6167 Read 6 Done New Done false 5169 Acked 6168 Read 10 Done New Done false 5170 Acked 6169 Read 2 Done New Done false 5171 Acked 6170 Flush 0 Done New Done false 5172 Acked 6171 Write 3 Done New Done false 5173 Acked 6172 Flush 0 Done New Done false 5174 Acked 6173 Write 5 Done New Done false 5175 Acked 6174 Write 6 Done New Done false 5176 Acked 6175 Flush 0 Done New Done false 5177 Acked 6176 Flush 0 Done New Done false 5178 Acked 6177 Write 1 Done New Done false 5179 Acked 6178 Write 1 Done New Done false 5180 Acked 6179 Write 4 Done New Done false 5181 Acked 6180 Read 1 Done New Done false 5182 Acked 6181 Write 8 Done New Done false 5183 Acked 6182 Read 9 Done New Done false 5184 Acked 6183 Write 1 Done New Done false 5185 Acked 6184 Write 10 Done New Done false 5186 Acked 6185 Write 10 Done New Done false 5187 Acked 6186 Flush 0 Done New Done false 5188 Acked 6187 Write 4 Done New Done false 5189 Acked 6188 Write 3 Done New Done false 5190 Acked 6189 Read 1 Done New Done false 5191 Acked 6190 Read 3 Done New Done false 5192 Acked 6191 Write 3 Done New Done false 5193 Acked 6192 Read 7 Done New Done false 5194 Acked 6193 Write 10 Done New Done false 5195 Acked 6194 Write 9 Done New Done false 5196 Acked 6195 Write 4 Done New Done false 5197 Acked 6196 Flush 0 Done New Done false 5198 Acked 6197 Write 4 Done New Done false 5199 Acked 6198 Write 1 Done New Done false 5200 Acked 6199 Read 10 Done New Done false 5201 Acked 6200 Write 7 Done New Done false 5202 Acked 6201 Read 1 Done New Done false 5203 Acked 6202 Write 7 Done New Done false 5204 Acked 6203 Write 8 Done New Done false 5205 Acked 6204 Write 2 Done New Done false 5206 Acked 6205 Write 7 Done New Done false 5207 Acked 6206 Write 4 Done New Done false 5208 Acked 6207 Read 2 Done New Done false 5209 Acked 6208 Write 1 Done New Done false 5210 Acked 6209 Write 6 Done New Done false 5211 Acked 6210 Write 4 Done New Done false 5212 Acked 6211 Read 10 Done New Done false 5213 Acked 6212 Read 1 Done New Done false 5214 Acked 6213 Write 6 Done New Done false 5215 Acked 6214 Write 4 Done New Done false 5216 Acked 6215 Read 9 Done New Done false 5217 Acked 6216 Read 1 Done New Done false 5218 Acked 6217 Write 3 Done New Done false 5219 Acked 6218 Read 1 Done New Done false 5220 Acked 6219 Read 1 Done New Done false 5221 Acked 6220 Write 1 Done New Done false 5222 Acked 6221 Read 2 Done New Done false 5223 Acked 6222 Read 9 Done New Done false 5224 Acked 6223 Read 5 Done New Done false 5225 Acked 6224 Write 4 Done New Done false 5226 Acked 6225 Read 1 Done New Done false 5227 Acked 6226 Read 7 Done New Done false 5228 Acked 6227 Read 8 Done New Done false 5229 Acked 6228 Read 4 Done New Done false 5230 Acked 6229 Read 7 Done New Done false 5231 Acked 6230 Flush 0 Done New Done false 5232 Acked 6231 Read 7 Done New Done false 5233 Acked 6232 Read 7 Done New Done false 5234 Acked 6233 Write 9 Done New Done false 5235 Acked 6234 Write 7 Done New Done false 5236 Acked 6235 Write 1 Done New Done false 5237 Acked 6236 Read 1 Done New Done false 5238 Acked 6237 Write 5 Done New Done false 5239 Acked 6238 Write 9 Done New Done false 5240 Acked 6239 Write 9 Done New Done false 5241 Acked 6240 Read 6 Done New Done false 5242 Acked 6241 Write 6 Done New Done false 5243 Acked 6242 Write 10 Done New Done false 5244 Acked 6243 Write 6 Done New Done false 5245 Acked 6244 Flush 0 Done New Done false 5246 Acked 6245 Read 5 Done New Done false 5247 Acked 6246 Flush 0 Done New Done false 5248 Acked 6247 Read 4 Done New Done false 5249 Acked 6248 Write 5 Done New Done false 5250 Acked 6249 Read 2 Done New Done false 5251 Acked 6250 Write 9 Done New Done false 5252 Acked 6251 Write 5 Done New Done false 5253 Acked 6252 Write 3 Done New Done false 5254 Acked 6253 Read 2 Done New Done false 5255 Acked 6254 Write 10 Done New Done false 5256 Acked 6255 Read 5 Done New Done false 5257 Acked 6256 Write 4 Done New Done false 5258 Acked 6257 Write 5 Done New Done false 5259 Acked 6258 Read 8 Done New Done false 5260 Acked 6259 Read 1 Done New Done false 5261 Acked 6260 Read 10 Done New Done false 5262 Acked 6261 Flush 0 Done New Done false 5263 Acked 6262 Read 1 Done New Done false 5264 Acked 6263 Write 10 Done New Done false 5265 Acked 6264 Write 7 Done New Done false 5266 Acked 6265 Read 5 Done New Done false 5267 Acked 6266 Flush 0 Done New Done false 5268 Acked 6267 Write 7 Done New Done false 5269 Acked 6268 Read 6 Done New Done false 5270 Acked 6269 Read 4 Done New Done false 5271 Acked 6270 Write 10 Done New Done false 5272 Acked 6271 Write 7 Done New Done false 5273 Acked 6272 Write 2 Done New Done false 5274 Acked 6273 Write 9 Done New Done false 5275 Acked 6274 Write 2 Done New Done false 5276 Acked 6275 Read 4 Done New Done false 5277 Acked 6276 Write 5 Done New Done false 5278 Acked 6277 Read 2 Done New Done false 5279 Acked 6278 Write 3 Done New Done false 5280 Acked 6279 Write 7 Done New Done false 5281 Acked 6280 Write 9 Done New Done false 5282 Acked 6281 Read 5 Done New Done false 5283 Acked 6282 Read 7 Done New Done false 5284 Acked 6283 Write 6 Done New Done false 5285 Acked 6284 Write 10 Done New Done false 5286 Acked 6285 Flush 0 Done New Done false 5287 Acked 6286 Write 9 Done New Done false 5288 Acked 6287 Flush 0 Done New Done false 5289 Acked 6288 Read 9 Done New Done false 5290 Acked 6289 Read 5 Done New Done false 5291 Acked 6290 Write 10 Done New Done false 5292 Acked 6291 Write 5 Done New Done false 5293 Acked 6292 Write 7 Done New Done false 5294 Acked 6293 Write 2 Done New Done false 5295 Acked 6294 Write 9 Done New Done false 5296 Acked 6295 Read 7 Done New Done false 5297 Acked 6296 Write 5 Done New Done false 5298 Acked 6297 Read 8 Done New Done false 5299 Acked 6298 Read 3 Done New Done false 5300 Acked 6299 Write 3 Done New Done false 5301 Acked 6300 Read 3 Done New Done false 5302 Acked 6301 Write 9 Done New Done false 5303 Acked 6302 Write 2 Done New Done false 5304 Acked 6303 Write 4 Done New Done false 5305 Acked 6304 Read 9 Done New Done false 5306 Acked 6305 Read 7 Done New Done false 5307 Acked 6306 Read 8 Done New Done false 5308 Acked 6307 Flush 0 Done New Done false 5309 Acked 6308 Write 5 Done New Done false 5310 Acked 6309 Read 3 Done New Done false 5311 Acked 6310 Read 9 Done New Done false 5312 Acked 6311 Read 2 Done New Done false 5313 Acked 6312 Flush 0 Done New Done false 5314 Acked 6313 Read 9 Done New Done false 5315 Acked 6314 Read 2 Done New Done false 5316 Acked 6315 Write 8 Done New Done false 5317 Acked 6316 Write 7 Done New Done false 5318 Acked 6317 Write 3 Done New Done false 5319 Acked 6318 Write 5 Done New Done false 5320 Acked 6319 Read 3 Done New Done false 5321 Acked 6320 Write 5 Done New Done false 5322 Acked 6321 Read 5 Done New Done false 5323 Acked 6322 Write 2 Done New Done false 5324 Acked 6323 Read 1 Done New Done false 5325 Acked 6324 Flush 0 Done New Done false 5326 Acked 6325 Read 1 Done New Done false 5327 Acked 6326 Read 1 Done New Done false 5328 Acked 6327 Write 10 Done New Done false 5329 Acked 6328 Write 5 Done New Done false 5330 Acked 6329 Write 7 Done New Done false 5331 Acked 6330 Write 6 Done New Done false 5332 Acked 6331 Write 3 Done New Done false 5333 Acked 6332 Read 9 Done New Done false 5334 Acked 6333 Read 8 Done New Done false 5335 Acked 6334 Read 6 Done New Done false 5336 Acked 6335 Write 5 Done New Done false 5337 Acked 6336 Read 2 Done New Done false 5338 Acked 6337 Write 5 Done New Done false 5339 Acked 6338 Write 2 Done New Done false 5340 Acked 6339 Read 3 Done New Done false 5341 Acked 6340 Read 7 Done New Done false 5342 Acked 6341 Read 9 Done New Done false 5343 Acked 6342 Write 10 Done New Done false 5344 Acked 6343 Read 1 Done New Done false 5345 Acked 6344 Read 9 Done New Done false 5346 Acked 6345 Write 8 Done New Done false 5347 Acked 6346 Read 9 Done New Done false 5348 Acked 6347 Read 6 Done New Done false 5349 Acked 6348 Read 5 Done New Done false 5350 Acked 6349 Flush 0 Done New Done false 5351 Acked 6350 Read 3 Done New Done false 5352 Acked 6351 Read 9 Done New Done false 5353 Acked 6352 Read 8 Done New Done false 5354 Acked 6353 Write 6 Done New Done false 5355 Acked 6354 Write 4 Done New Done false 5356 Acked 6355 Read 5 Done New Done false 5357 Acked 6356 Write 7 Done New Done false 5358 Acked 6357 Read 5 Done New Done false 5359 Acked 6358 Read 9 Done New Done false 5360 Acked 6359 Read 1 Done New Done false 5361 Acked 6360 Read 4 Done New Done false 5362 Acked 6361 Read 6 Done New Done false 5363 Acked 6362 Write 4 Done New Done false 5364 Acked 6363 Read 9 Done New Done false 5365 Acked 6364 Read 2 Done New Done false 5366 Acked 6365 Write 5 Done New Done false 5367 Acked 6366 Write 9 Done New Done false 5368 Acked 6367 Flush 0 Done New Done false 5369 Acked 6368 Write 6 Done New Done false 5370 Acked 6369 Read 2 Done New Done false 5371 Acked 6370 Write 6 Done New Done false 5372 Acked 6371 Write 5 Done New Done false 5373 Acked 6372 Write 3 Done New Done false 5374 Acked 6373 Read 4 Done New Done false 5375 Acked 6374 Write 9 Done New Done false 5376 Acked 6375 Write 4 Done New Done false 5377 Acked 6376 Read 8 Done New Done false 5378 Acked 6377 Read 8 Done New Done false 5379 Acked 6378 Read 7 Done New Done false 5380 Acked 6379 Write 10 Done New Done false 5381 Acked 6380 Read 2 Done New Done false 5382 Acked 6381 Flush 0 Done New Done false 5383 Acked 6382 Read 4 Done New Done false 5384 Acked 6383 Write 1 Done New Done false 5385 Acked 6384 Read 9 Done New Done false 5386 Acked 6385 Read 3 Done New Done false 5387 Acked 6386 Write 5 Done New Done false 5388 Acked 6387 Read 4 Done New Done false 5389 Acked 6388 Read 6 Done New Done false 5390 Acked 6389 Read 5 Done New Done false 5391 Acked 6390 Read 5 Done New Done false 5392 Acked 6391 Read 6 Done New Done false 5393 Acked 6392 Write 8 Done New Done false 5394 Acked 6393 Read 9 Done New Done false 5395 Acked 6394 Read 2 Done New Done false 5396 Acked 6395 Write 3 Done New Done false 5397 Acked 6396 Read 9 Done New Done false 5398 Acked 6397 Write 6 Done New Done false 5399 Acked 6398 Write 10 Done New Done false 5400 Acked 6399 Write 2 Done New Done false 5401 Acked 6400 Write 3 Done New Done false 5402 Acked 6401 Read 1 Done New Done false 5403 Acked 6402 Read 4 Done New Done false 5404 Acked 6403 Read 9 Done New Done false 5405 Acked 6404 Write 6 Done New Done false 5406 Acked 6405 Read 3 Done New Done false 5407 Acked 6406 Write 4 Done New Done false 5408 Acked 6407 Read 1 Done New Done false 5409 Acked 6408 Read 1 Done New Done false 5410 Acked 6409 Write 3 Done New Done false 5411 Acked 6410 Read 6 Done New Done false 5412 Acked 6411 Write 6 Done New Done false 5413 Acked 6412 Read 6 Done New Done false 5414 Acked 6413 Write 9 Done New Done false 5415 Acked 6414 Write 1 Done New Done false 5416 Acked 6415 Write 3 Done New Done false 5417 Acked 6416 Read 1 Done New Done false 5418 Acked 6417 Read 8 Done New Done false 5419 Acked 6418 Flush 0 Done New Done false 5420 Acked 6419 Read 1 Done New Done false 5421 Acked 6420 Read 8 Done New Done false 5422 Acked 6421 Write 6 Done New Done false 5423 Acked 6422 Write 1 Done New Done false 5424 Acked 6423 Write 2 Done New Done false 5425 Acked 6424 Read 5 Done New Done false 5426 Acked 6425 Write 1 Done New Done false 5427 Acked 6426 Read 7 Done New Done false 5428 Acked 6427 Write 1 Done New Done false 5429 Acked 6428 Read 2 Done New Done false 5430 Acked 6429 Read 10 Done New Done false 5431 Acked 6430 Write 10 Done New Done false 5432 Acked 6431 Read 7 Done New Done false 5433 Acked 6432 Write 3 Done New Done false 5434 Acked 6433 Read 2 Done New Done false 5435 Acked 6434 Write 1 Done New Done false 5436 Acked 6435 Write 10 Done New Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 300 0 300 Sent 0 0 0 0 Done 300 0 300 600 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6418 6135 6418 Downstairs last five completed: 6135 6134 6133 6132 6131 Upstairs last five completed: 5436 5435 5434 5433 5432 CLIENT: Up:0 ds:300 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:09.319857592Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:09.319925618Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:09.31995441Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:09.319981543Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:09.320012461Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:09.320036262Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:09.320257315Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:10.322106527Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:10.322192327Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:10.322236267Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:10.322282725Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:10.322316564Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:10.322345312Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:10.322523213Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1203) [8820][1] reports Running {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.323148863Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.323309498Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.325324068Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.325362334Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.32538669Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 6135","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.325687307Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.325717477Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.325740639Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.325762675Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:11.325825598Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 6436 6435 6434 6433 6432 Upstairs last five completed: 5437 5436 5435 5434 5433 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:34:12.874 INFO request completed, latency_us: 478, response_code: 204[8810] Got stop action so:false kr:false , uri: /stop/cid/0, method: POST, req_id: 7a5c333a-55ed-41c6-a1d8-da64d4c310d3, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:12.875555159Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:12.875629371Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:12.87567393Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:12.875697446Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:34:12.875 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/0, method: GET, req_id: da057ab6-94f6-4656-80e6-88baccf53091, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"[0] client re-new 0 jobs since flush 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:12.875974675Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:12.876035398Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:13.878077342Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:13.878169239Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:13.878197684Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:13.878224317Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:13.878244759Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:13.878268648Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:13.878316238Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:14.879277Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:14.879346104Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:14.879374276Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:14.879401149Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:14.879421549Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:14.879453417Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:14.879499043Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:15.880428207Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:15.880500584Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:15.880528839Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:15.880559612Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:15.880580818Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:15.880603306Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:15.880713429Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:34:16.878 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/0, method: GET, req_id: dd474554-3c8e-46aa-8578-f50cbdc54db6, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 50 len 40960 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:16.881283491Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:16.881349137Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:16.881377497Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:16.881409382Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:16.882817779Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:16.882845812Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 1 jobs since flush 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:16.88286946Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:16.883104891Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 002/300 Flush 003/300 Write block 2680 len 20480 data: 2 2 3 3 3 004/300 Read block 700 len 28672 005/300 Flush 006/300 Write block 1089 len 24576 data: 4 4 4 4 4 5 007/300 Write block 1240 len 12288 data: 4 4 5 008/300 Read block 186 len 32768 009/300 Flush 010/300 Flush 011/300 Read block 1244 len 40960 012/300 Read block 3572 len 8192 013/300 Read block 1507 len 16384 014/300 Write block 2867 len 40960 data: 4 4 4 4 5 3 3 3 2 2 015/300 Read block 4956 len 12288 016/300 Write block 1895 len 20480 data: 4 4 3 2 2 017/300 Read block 2802 len 40960 018/300 Write block 3645 len 28672 data: 2 2 3 3 4 4 3 019/300 Read block 3884 len 4096 020/300 Read block 156 len 32768 021/300 Read block 282 len 16384 022/300 Write block 679 len 8192 data: 3 2 023/300 Read block 634 len 28672 024/300 Write block 4962 len 12288 data: 3 4 4 025/300 Read block 1929 len 32768 026/300 Write block 2838 len 16384 data: 6 6 6 4 027/300 Read block 1128 len 8192 028/300 Read block 691 len 8192 029/300 Write block 2014 len 40960 data: 3 4 4 4 3 2 2 2 2 2 030/300 Read block 1071 len 40960 031/300 Write block 3641 len 8192 data: 3 2 032/300 Read block 852 len 16384 033/300 Read block 3572 len 28672 034/300 Read block 4311 len 36864 035/300 Read block 440 len 16384 036/300 Flush 037/300 Read block 165 len 12288 038/300 Write block 1704 len 32768 data: 5 6 6 5 2 3 4 5 039/300 Write block 1694 len 8192 data: 6 6 040/300 Read block 3772 len 28672 041/300 Read block 3665 len 16384 042/300 Read block 3930 len 16384 043/300 Read block 4444 len 28672 044/300 Read block 3265 len 12288 045/300 Read block 2034 len 20480 046/300 Read block 1131 len 28672 047/300 Flush 048/300 Read block 1181 len 12288 049/300 Write block 2632 len 24576 data: 5 5 5 8 7 7 050/300 Read block 3028 len 24576 051/300 Flush 052/300 Read block 2200 len 24576 053/300 Write block 516 len 20480 data: 6 4 5 4 4 054/300 Read block 3130 len 20480 055/300 Write block 432 len 16384 data: 6 5 5 5 056/300 Write block 2107 len 20480 data: 3 3 3 4 3 057/300 Write block 3227 len 32768 data: 4 5 6 6 5 5 5 5 058/300 Read block 213 len 16384 059/300 Write block 2 len 40960 data: 3 3 4 3 5 4 5 6 8 10 060/300 Read block 1684 len 12288 061/300 Read block 1116 len 32768 062/300 Read block 4093 len 16384 063/300 Read block 1311 len 16384 064/300 Read block 2065 len 12288 065/300 Write block 3120 len 24576 data: 4 4 4 4 4 5 066/300 Write block 1405 len 24576 data: 4 2 2 2 2 2 067/300 Read block 3257 len 28672 068/300 Write block 3412 len 4096 data: 5 069/300 Read block 1648 len 36864 070/300 Read block 3862 len 32768 071/300 Write block 939 len 32768 data: 7 5 5 5 5 4 5 6 072/300 Write block 4213 len 40960 data: 5 6 8 6 6 7 6 7 6 6 073/300 Write block 2074 len 28672 data: 2 2 3 4 4 3 2 074/300 Write block 932 len 4096 data: 4 075/300 Flush 076/300 Write block 1686 len 36864 data: 6 7 7 5 5 5 4 4 7 077/300 Write block 2151 len 32768 data: 4 5 5 6 6 6 6 8 078/300 Read block 1516 len 28672 079/300 Read block 2209 len 28672 080/300 Read block 3025 len 8192 081/300 Write block 855 len 16384 data: 3 3 3 3 082/300 Write block 1635 len 4096 data: 6 083/300 Write block 265 len 40960 data: 5 5 7 8 7 7 8 8 7 6 084/300 Read block 4724 len 16384 085/300 Read block 2283 len 4096 086/300 Read block 1195 len 36864 087/300 Read block 3318 len 20480 088/300 Read block 3977 len 8192 089/300 Read block 4596 len 32768 090/300 Read block 3510 len 40960 091/300 Read block 4796 len 36864 092/300 Read block 4413 len 16384 093/300 Write block 2716 len 24576 data: 5 4 3 3 3 3 094/300 Read block 3831 len 16384 095/300 Read block 705 len 20480 096/300 Read block 1232 len 16384 097/300 Write block 1579 len 20480 data: 2 2 4 4 4 098/300 Read block 893 len 24576 099/300 Write block 2773 len 28672 data: 9 5 5 5 4 3 3 100/300 Read block 4013 len 40960 101/300 Write block 1446 len 40960 data: 3 3 3 3 4 3 2 2 2 2 102/300 Read block 4750 len 24576 103/300 Read block 2620 len 32768 104/300 Read block 136 len 4096 105/300 Read block 2742 len 8192 106/300 Write block 3599 len 12288 data: 4 5 5 107/300 Write block 4416 len 36864 data: 2 2 4 5 5 5 6 6 5 108/300 Write block 4285 len 24576 data: 4 4 3 2 2 4 109/300 Read block 3344 len 12288 110/300 Read block 4868 len 4096 111/300 Write block 4888 len 8192 data: 7 4 112/300 Read block 256 len 4096 113/300 Read block 1186 len 4096 114/300 Read block 1833 len 8192 115/300 Write block 3169 len 16384 data: 2 2 2 3 116/300 Write block 4591 len 28672 data: 5 4 2 2 3 3 3 117/300 Read block 158 len 40960 118/300 Write block 3275 len 16384 data: 2 2 3 3 119/300 Read block 3979 len 12288 120/300 Read block 2331 len 32768 121/300 Write block 391 len 24576 data: 5 6 7 8 8 5 122/300 Flush 123/300 Read block 322 len 16384 124/300 Read block 3388 len 4096 125/300 Write block 3190 len 32768 data: 4 4 4 5 5 5 4 4 126/300 Write block 2599 len 40960 data: 4 3 2 2 3 3 4 4 4 5 127/300 Read block 3960 len 4096 128/300 Read block 1130 len 12288 129/300 Write block 3331 len 24576 data: 4 7 6 5 5 5 130/300 Read block 1788 len 28672 131/300 Write block 4951 len 4096 data: 2 132/300 Read block 2691 len 40960 133/300 Read block 4564 len 4096 134/300 Read block 3272 len 36864 135/300 Write block 3924 len 28672 data: 6 6 7 7 6 5 6 136/300 Write block 4803 len 16384 data: 7 7 8 7 137/300 Write block 2077 len 32768 data: 5 5 4 3 2 2 3 3 138/300 Write block 1570 len 28672 data: 3 3 3 3 3 3 2 139/300 Read block 952 len 4096 140/300 Read block 1093 len 4096 141/300 Flush 142/300 Read block 1516 len 4096 143/300 Write block 1957 len 36864 data: 3 3 3 2 2 2 4 4 3 144/300 Read block 2845 len 16384 145/300 Read block 73 len 4096 146/300 Read block 4427 len 20480 147/300 Read block 4990 len 20480 148/300 Read block 3372 len 40960 149/300 Read block 1551 len 36864 150/300 Read block 562 len 32768 151/300 Read block 1747 len 28672 152/300 Write block 2374 len 32768 data: 5 4 4 4 3 3 3 2 153/300 Flush 154/300 Read block 205 len 8192 155/300 Read block 3872 len 24576 156/300 Write block 4884 len 20480 data: 5 8 7 9 8 157/300 Write block 2172 len 16384 data: 2 2 2 2 158/300 Read block 3884 len 12288 159/300 Write block 2032 len 28672 data: 7 7 5 5 6 5 5 160/300 Write block 3098 len 16384 data: 3 4 4 4 161/300 Read block 607 len 16384 162/300 Write block 4471 len 20480 data: 3 3 3 4 4 163/300 Read block 1310 len 20480 164/300 Read block 2301 len 36864 165/300 Write block 4926 len 28672 data: 7 8 8 6 5 7 6 166/300 Flush 167/300 Write block 822 len 32768 data: 4 4 4 4 4 3 3 5 168/300 Flush 169/300 Read block 4600 len 40960 170/300 Write block 2242 len 20480 data: 3 2 2 3 3 171/300 Read block 4247 len 4096 172/300 Read block 1121 len 4096 173/300 Read block 2002 len 16384 174/300 Read block 658 len 32768 175/300 Write block 1806 len 8192 data: 3 3 176/300 Read block 1731 len 32768 177/300 Flush 178/300 Read block 1934 len 20480 179/300 Read block 3553 len 4096 180/300 Read block 4191 len 32768 181/300 Write block 656 len 36864 data: 7 7 8 9 10 9 9 11 11 182/300 Read block 4755 len 24576 183/300 Read block 1201 len 12288 184/300 Read block 363 len 12288 185/300 Read block 350 len 8192 186/300 Write block 4265 len 24576 data: 6 8 7 10 10 8 187/300 Write block 2544 len 12288 data: 3 3 4 188/300 Write block 4901 len 8192 data: 4 5 189/300 Read block 4140 len 36864 190/300 Write block 1872 len 8192 data: 4 4 191/300 Write block 1265 len 32768 data: 6 5 4 3 3 4 4 4 192/300 Write block 722 len 24576 data: 3 2 3 5 5 5 193/300 Read block 914 len 20480 194/300 Write block 2234 len 28672 data: 4 6 6 5 5 5 6 195/300 Write block 2854 len 40960 data: 5 5 5 6 6 6 5 4 3 4 196/300 Read block 1637 len 36864 197/300 Write block 2862 len 28672 data: 4 5 4 4 4 5 5 198/300 Write block 2736 len 8192 data: 5 5 199/300 Read block 2824 len 12288 200/300 Read block 1403 len 12288 201/300 Read block 3995 len 4096 202/300 Read block 1462 len 8192 203/300 Read block 1224 len 4096 204/300 Flush 205/300 Read block 4149 len 8192 206/300 Write block 573 len 32768 data: 4 3 3 4 5 5 5 5 207/300 Write block 2445 len 12288 data: 5 6 6 208/300 Write block 2160 len 28672 data: 6 5 5 4 6 6 5 209/300 Write block 1840 len 36864 data: 2 2 2 3 3 3 3 3 3 210/300 Read block 3886 len 32768 211/300 Flush 212/300 Write block 3452 len 24576 data: 4 4 5 6 5 5 213/300 Read block 1473 len 8192 214/300 Write block 3227 len 8192 data: 5 6 215/300 Write block 1920 len 24576 data: 5 2 2 2 3 4 216/300 Read block 1728 len 36864 217/300 Write block 286 len 40960 data: 5 5 4 4 4 4 5 5 6 6 218/300 Read block 4068 len 8192 219/300 Flush 220/300 Write block 418 len 20480 data: 6 6 6 7 7 221/300 Read block 1125 len 32768 222/300 Read block 3356 len 16384 223/300 Read block 3332 len 28672 224/300 Read block 4350 len 20480 225/300 Write block 3383 len 32768 data: 2 4 3 3 3 3 3 3 226/300 Flush 227/300 Flush 228/300 Write block 512 len 40960 data: 3 4 5 5 7 5 6 5 5 4 229/300 Flush 230/300 Write block 858 len 12288 data: 4 4 5 231/300 Read block 3684 len 20480 232/300 Write block 1467 len 28672 data: 4 4 4 3 3 2 2 233/300 Read block 1773 len 24576 234/300 Flush 235/300 Read block 3129 len 12288 236/300 Read block 2234 len 36864 237/300 Write block 3288 len 8192 data: 2 2 238/300 Read block 1576 len 4096 239/300 Flush 240/300 Write block 3800 len 24576 data: 5 4 3 3 3 3 241/300 Write block 2363 len 4096 data: 5 242/300 Flush 243/300 Write block 2074 len 8192 data: 3 3 244/300 Read block 3622 len 28672 245/300 Flush 246/300 Write block 3365 len 20480 data: 4 4 4 4 3 247/300 Read block 2448 len 28672 248/300 Write block 902 len 40960 data: 3 4 3 3 3 3 3 3 2 3 249/300 Write block 2785 len 24576 data: 4 4 5 5 5 4 250/300 Read block 4006 len 20480 251/300 Flush 252/300 Write block 9 len 24576 data: 7 9 11 8 8 8 253/300 Flush 254/300 Write block 4110 len 28672 data: 2 2 2 3 3 2 2 255/300 Write block 3643 len 12288 data: 2 2 3 256/300 Write block 342 len 8192 data: 5 4 257/300 Write block 2609 len 36864 data: 6 7 4 4 6 6 7 7 6 258/300 Read block 1613 len 20480 259/300 Read block 2195 len 36864 260/300 Write block 364 len 4096 data: 7 261/300 Flush 262/300 Flush 263/300 Read block 4922 len 40960 264/300 Read block 4483 len 12288 265/300 Flush 266/300 Read block 155 len 20480 267/300 Read block 2722 len 24576 268/300 Write block 3515 len 4096 data: 4 269/300 Read block 1753 len 8192 270/300 Write block 3178 len 36864 data: 6 6 6 5 5 5 5 3 4 271/300 Write block 1669 len 28672 data: 6 5 5 4 4 4 6 272/300 Write block 4726 len 4096 data: 4 273/300 Write block 362 len 28672 data: 5 6 8 6 7 8 5 274/300 Flush 275/300 Read block 4081 len 20480 276/300 Write block 3553 len 12288 data: 4 3 3 277/300 Write block 2152 len 4096 data: 6 278/300 Read block 2772 len 32768 279/300 Write block 2539 len 32768 data: 3 4 3 3 3 4 4 5 280/300 Read block 2252 len 20480 281/300 Write block 283 len 20480 data: 6 5 4 6 6 282/300 Write block 4622 len 28672 data: 4 4 3 4 7 6 5 283/300 Read block 2399 len 36864 284/300 Read block 947 len 20480 285/300 Read block 2713 len 40960 286/300 Read block 1402 len 4096 287/300 Write block 3993 len 16384 data: 9 8 8 9 288/300 Write block 3440 len 20480 data: 3 2 2 2 2 289/300 Write block 3943 len 4096 data: 2 290/300 Write block 4087 len 24576 data: 5 5 4 6 6 5 291/300 Read block 4758 len 40960 292/300 Read block 1131 len 16384 293/300 Flush 294/300 Read block 1808 len 16384 295/300 Read block 3237 len 24576 296/300 Read block 1917 len 8192 297/300 Read block 575 len 28672 298/300 Read block 4977 len 8192 299/300 Read block 464 len 36864 300/300 Write block 4963 len 36864 data: 5 5 5 3 3 3 4 4 3 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:34:17.470 INFO request completed, latency_us: 297, response_code: 204, uri: /start/cid/0, method: POST, req_id: 97725617-6133-482c-826a-ac397d35629b, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 5438 Acked 6437 Read 10 New Done Done true 5439 Acked 6438 Flush 0 New Done Done false 5440 Acked 6439 Write 5 New Done Done false 5441 Acked 6440 Read 7 New Done Done false 5442 Acked 6441 Flush 0 New Done Done false 5443 Acked 6442 Write 6 New Done Done false 5444 Acked 6443 Write 3 New Done Done false 5445 Acked 6444 Read 8 New Done Done false 5446 Acked 6445 Flush 0 New Done Done false 5447 Acked 6446 Flush 0 New Done Done false 5448 Acked 6447 Read 10 New Done Done false 5449 Acked 6448 Read 2 New Done Done false 5450 Acked 6449 Read 4 New Done Done false 5451 Acked 6450 Write 10 New Done Done false 5452 Acked 6451 Read 3 New Done Done false 5453 Acked 6452 Write 5 New Done Done false 5454 Acked 6453 Read 10 New Done Done false 5455 Acked 6454 Write 7 New Done Done false 5456 Acked 6455 Read 1 New Done Done false 5457 Acked 6456 Read 8 New Done Done false 5458 Acked 6457 Read 4 New Done Done false 5459 Acked 6458 Write 2 New Done Done false 5460 Acked 6459 Read 7 New Done Done false 5461 Acked 6460 Write 3 New Done Done false 5462 Acked 6461 Read 8 New Done Done false 5463 Acked 6462 Write 4 New Done Done false 5464 Acked 6463 Read 2 New Done Done false 5465 Acked 6464 Read 2 New Done Done false 5466 Acked 6465 Write 10 New Done Done false 5467 Acked 6466 Read 10 New Done Done false 5468 Acked 6467 Write 2 New Done Done false 5469 Acked 6468 Read 4 New Done Done false 5470 Acked 6469 Read 7 New Done Done false 5471 Acked 6470 Read 9 New Done Done false 5472 Acked 6471 Read 4 New Done Done false 5473 Acked 6472 Flush 0 New Done Done false 5474 Acked 6473 Read 3 New Done Done false 5475 Acked 6474 Write 8 New Done Done false 5476 Acked 6475 Write 2 New Done Done false 5477 Acked 6476 Read 7 New Done Done false 5478 Acked 6477 Read 4 New Done Done false 5479 Acked 6478 Read 4 New Done Done false 5480 Acked 6479 Read 7 New Done Done false 5481 Acked 6480 Read 3 New Done Done false 5482 Acked 6481 Read 5 New Done Done false 5483 Acked 6482 Read 7 New Done Done false 5484 Acked 6483 Flush 0 New Done Done false 5485 Acked 6484 Read 3 New Done Done false 5486 Acked 6485 Write 6 New Done Done false 5487 Acked 6486 Read 6 New Done Done false 5488 Acked 6487 Flush 0 New Done Done false 5489 Acked 6488 Read 6 New Done Done false 5490 Acked 6489 Write 5 New Done Done false 5491 Acked 6490 Read 5 New Done Done false 5492 Acked 6491 Write 4 New Done Done false 5493 Acked 6492 Write 5 New Done Done false 5494 Acked 6493 Write 8 New Done Done false 5495 Acked 6494 Read 4 New Done Done false 5496 Acked 6495 Write 10 New Done Done false 5497 Acked 6496 Read 3 New Done Done false 5498 Acked 6497 Read 8 New Done Done false 5499 Acked 6498 Read 4 New Done Done false 5500 Acked 6499 Read 4 New Done Done false 5501 Acked 6500 Read 3 New Done Done false 5502 Acked 6501 Write 6 New Done Done false 5503 Acked 6502 Write 6 New Done Done false 5504 Acked 6503 Read 7 New Done Done false 5505 Acked 6504 Write 1 New Done Done false 5506 Acked 6505 Read 9 New Done Done false 5507 Acked 6506 Read 8 New Done Done false 5508 Acked 6507 Write 8 New Done Done false 5509 Acked 6508 Write 10 New Done Done false 5510 Acked 6509 Write 7 New Done Done false 5511 Acked 6510 Write 1 New Done Done false 5512 Acked 6511 Flush 0 New Done Done false 5513 Acked 6512 Write 9 New Done Done false 5514 Acked 6513 Write 8 New Done Done false 5515 Acked 6514 Read 7 New Done Done false 5516 Acked 6515 Read 7 New Done Done false 5517 Acked 6516 Read 2 New Done Done false 5518 Acked 6517 Write 4 New Done Done false 5519 Acked 6518 Write 1 New Done Done false 5520 Acked 6519 Write 10 New Done Done false 5521 Acked 6520 Read 4 New Done Done false 5522 Acked 6521 Read 1 New Done Done false 5523 Acked 6522 Read 9 New Done Done false 5524 Acked 6523 Read 5 New Done Done false 5525 Acked 6524 Read 2 New Done Done false 5526 Acked 6525 Read 8 New Done Done false 5527 Acked 6526 Read 10 New Done Done false 5528 Acked 6527 Read 9 New Done Done false 5529 Acked 6528 Read 4 New Done Done false 5530 Acked 6529 Write 6 New Done Done false 5531 Acked 6530 Read 4 New Done Done false 5532 Acked 6531 Read 5 New Done Done false 5533 Acked 6532 Read 4 New Done Done false 5534 Acked 6533 Write 5 New Done Done false 5535 Acked 6534 Read 6 New Done Done false 5536 Acked 6535 Write 7 New Done Done false 5537 Acked 6536 Read 10 New Done Done false 5538 Acked 6537 Write 10 New Done Done false 5539 Acked 6538 Read 6 New Done Done false 5540 Acked 6539 Read 8 New Done Done false 5541 Acked 6540 Read 1 New Done Done false 5542 Acked 6541 Read 2 New Done Done false 5543 Acked 6542 Write 3 New Done Done false 5544 Acked 6543 Write 9 New Done Done false 5545 Acked 6544 Write 6 New Done Done false 5546 Acked 6545 Read 3 New Done Done false 5547 Acked 6546 Read 1 New Done Done false 5548 Acked 6547 Write 2 New Done Done false 5549 Acked 6548 Read 1 New Done Done false 5550 Acked 6549 Read 1 New Done Done false 5551 Acked 6550 Read 2 New Done Done false 5552 Acked 6551 Write 4 New Done Done false 5553 Acked 6552 Write 7 New Done Done false 5554 Acked 6553 Read 10 New Done Done false 5555 Acked 6554 Write 4 New Done Done false 5556 Acked 6555 Read 3 New Done Done false 5557 Acked 6556 Read 8 New Done Done false 5558 Acked 6557 Write 6 New Done Done false 5559 Acked 6558 Flush 0 New Done Done false 5560 Acked 6559 Read 4 New Done Done false 5561 Acked 6560 Read 1 New Done Done false 5562 Acked 6561 Write 8 New Done Done false 5563 Acked 6562 Write 10 New Done Done false 5564 Acked 6563 Read 1 New Done Done false 5565 Acked 6564 Read 3 New Done Done false 5566 Acked 6565 Write 6 New Done Done false 5567 Acked 6566 Read 7 New Done Done false 5568 Acked 6567 Write 1 New Done Done false 5569 Acked 6568 Read 10 New Done Done false 5570 Acked 6569 Read 1 New Done Done false 5571 Acked 6570 Read 9 New Done Done false 5572 Acked 6571 Write 7 New Done Done false 5573 Acked 6572 Write 4 New Done Done false 5574 Acked 6573 Write 8 New Done Done false 5575 Acked 6574 Write 7 New Done Done false 5576 Acked 6575 Read 1 New Done Done false 5577 Acked 6576 Read 1 New Done Done false 5578 Acked 6577 Flush 0 New Done Done false 5579 Acked 6578 Read 1 New Done Done false 5580 Acked 6579 Write 9 New Done Done false 5581 Acked 6580 Read 4 New Done Done false 5582 Acked 6581 Read 1 New Done Done false 5583 Acked 6582 Read 5 New Done Done false 5584 Acked 6583 Read 5 New Done Done false 5585 Acked 6584 Read 10 New Done Done false 5586 Acked 6585 Read 9 New Done Done false 5587 Acked 6586 Read 8 New Done Done false 5588 Acked 6587 Read 7 New Done Done false 5589 Acked 6588 Write 8 New Done Done false 5590 Acked 6589 Flush 0 New Done Done false 5591 Acked 6590 Read 2 New Done Done false 5592 Acked 6591 Read 6 New Done Done false 5593 Acked 6592 Write 5 New Done Done false 5594 Acked 6593 Write 4 New Done Done false 5595 Acked 6594 Read 3 New Done Done false 5596 Acked 6595 Write 7 New Done Done false 5597 Acked 6596 Write 4 New Done Done false 5598 Acked 6597 Read 4 New Done Done false 5599 Acked 6598 Write 5 New Done Done false 5600 Acked 6599 Read 5 New Done Done false 5601 Acked 6600 Read 9 New Done Done false 5602 Acked 6601 Write 7 New Done Done false 5603 Acked 6602 Flush 0 New Done Done false 5604 Acked 6603 Write 8 New Done Done false 5605 Acked 6604 Flush 0 New Done Done false 5606 Acked 6605 Read 10 New Done Done false 5607 Acked 6606 Write 5 New Done Done false 5608 Acked 6607 Read 1 New Done Done false 5609 Acked 6608 Read 1 New Done Done false 5610 Acked 6609 Read 4 New Done Done false 5611 Acked 6610 Read 8 New Done Done false 5612 Acked 6611 Write 2 New Done Done false 5613 Acked 6612 Read 8 New Done Done false 5614 Acked 6613 Flush 0 New Done Done false 5615 Acked 6614 Read 5 New Done Done false 5616 Acked 6615 Read 1 New Done Done false 5617 Acked 6616 Read 8 New Done Done false 5618 Acked 6617 Write 9 New Done Done false 5619 Acked 6618 Read 6 New Done Done false 5620 Acked 6619 Read 3 New Done Done false 5621 Acked 6620 Read 3 New Done Done false 5622 Acked 6621 Read 2 New Done Done false 5623 Acked 6622 Write 6 New Done Done false 5624 Acked 6623 Write 3 New Done Done false 5625 Acked 6624 Write 2 New Done Done false 5626 Acked 6625 Read 9 New Done Done false 5627 Acked 6626 Flush 0 New Done Done false 5628 Acked 6627 Write 2 New Done Done false 5629 Acked 6628 Write 8 New Done Done false 5630 Acked 6629 Write 6 New Done Done false 5631 Acked 6630 Read 5 New Done Done false 5632 Acked 6631 Write 7 New Done Done false 5633 Acked 6632 Write 10 New Done Done false 5634 Acked 6633 Read 9 New Done Done false 5635 Acked 6634 Write 7 New Done Done false 5636 Acked 6635 Write 2 New Done Done false 5637 Acked 6636 Read 3 New Done Done false 5638 Acked 6637 Read 3 New Done Done false 5639 Acked 6638 Read 1 New Done Done false 5640 Acked 6639 Read 2 New Done Done false 5641 Acked 6640 Read 1 New Done Done false 5642 Acked 6641 Flush 0 New Done Done false 5643 Acked 6642 Read 2 New Done Done false 5644 Acked 6643 Write 8 New Done Done false 5645 Acked 6644 Write 3 New Done Done false 5646 Acked 6645 Write 7 New Done Done false 5647 Acked 6646 Write 9 New Done Done false 5648 Acked 6647 Read 8 New Done Done false 5649 Acked 6648 Flush 0 New Done Done false 5650 Acked 6649 Write 6 New Done Done false 5651 Acked 6650 Read 2 New Done Done false 5652 Acked 6651 Write 2 New Done Done false 5653 Acked 6652 Write 6 New Done Done false 5654 Acked 6653 Read 9 New Done Done false 5655 Acked 6654 Write 10 New Done Done false 5656 Acked 6655 Read 2 New Done Done false 5657 Acked 6656 Flush 0 New Done Done false 5658 Acked 6657 Write 5 New Done Done false 5659 Acked 6658 Read 8 New Done Done false 5660 Acked 6659 Read 4 New Done Done false 5661 Acked 6660 Read 7 New Done Done false 5662 Acked 6661 Read 5 New Done Done false 5663 Acked 6662 Write 8 New Done Done false 5664 Acked 6663 Flush 0 New Done Done false 5665 Acked 6664 Flush 0 New Done Done false 5666 Acked 6665 Write 10 New Done Done false 5667 Acked 6666 Flush 0 New Done Done false 5668 Acked 6667 Write 3 New Done Done false 5669 Acked 6668 Read 5 New Done Done false 5670 Acked 6669 Write 7 New Done Done false 5671 Acked 6670 Read 6 New Done Done false 5672 Acked 6671 Flush 0 New Done Done false 5673 Acked 6672 Read 3 New Done Done false 5674 Acked 6673 Read 9 New Done Done false 5675 Acked 6674 Write 2 New Done Done false 5676 Acked 6675 Read 1 New Done Done false 5677 Acked 6676 Flush 0 New Done Done false 5678 Acked 6677 Write 6 New Done Done false 5679 Acked 6678 Write 1 New Done Done false 5680 Acked 6679 Flush 0 New Done Done false 5681 Acked 6680 Write 2 New Done Done false 5682 Acked 6681 Read 7 New Done Done false 5683 Acked 6682 Flush 0 New Done Done false 5684 Acked 6683 Write 5 New Done Done false 5685 Acked 6684 Read 7 New Done Done false 5686 Acked 6685 Write 10 New Done Done false 5687 Acked 6686 Write 6 New Done Done false 5688 Acked 6687 Read 5 New Done Done false 5689 Acked 6688 Flush 0 New Done Done false 5690 Acked 6689 Write 6 New Done Done false 5691 Acked 6690 Flush 0 New Done Done false 5692 Acked 6691 Write 7 New Done Done false 5693 Acked 6692 Write 3 New Done Done false 5694 Acked 6693 Write 2 New Done Done false 5695 Acked 6694 Write 9 New Done Done false 5696 Acked 6695 Read 5 New Done Done false 5697 Acked 6696 Read 9 New Done Done false 5698 Acked 6697 Write 1 New Done Done false 5699 Acked 6698 Flush 0 New Done Done false 5700 Acked 6699 Flush 0 New Done Done false 5701 Acked 6700 Read 10 New Done Done false 5702 Acked 6701 Read 3 New Done Done false 5703 Acked 6702 Flush 0 New Done Done false 5704 Acked 6703 Read 5 New Done Done false 5705 Acked 6704 Read 6 New Done Done false 5706 Acked 6705 Write 1 New Done Done false 5707 Acked 6706 Read 2 New Done Done false 5708 Acked 6707 Write 9 New Done Done false 5709 Acked 6708 Write 7 New Done Done false 5710 Acked 6709 Write 1 New Done Done false 5711 Acked 6710 Write 7 New Done Done false 5712 Acked 6711 Flush 0 New Done Done false 5713 Acked 6712 Read 5 New Done Done false 5714 Acked 6713 Write 3 New Done Done false 5715 Acked 6714 Write 1 New Done Done false 5716 Acked 6715 Read 8 New Done Done false 5717 Acked 6716 Write 8 New Done Done false 5718 Acked 6717 Read 5 New Done Done false 5719 Acked 6718 Write 5 New Done Done false 5720 Acked 6719 Write 7 New Done Done false 5721 Acked 6720 Read 9 New Done Done false 5722 Acked 6721 Read 5 New Done Done false 5723 Acked 6722 Read 10 New Done Done false 5724 Acked 6723 Read 1 New Done Done false 5725 Acked 6724 Write 4 New Done Done false 5726 Acked 6725 Write 5 New Done Done false 5727 Acked 6726 Write 1 New Done Done false 5728 Acked 6727 Write 6 New Done Done false 5729 Acked 6728 Read 10 New Done Done false 5730 Acked 6729 Read 4 New Done Done false 5731 Acked 6730 Flush 0 New Done Done false 5732 Acked 6731 Read 4 New Done Done false 5733 Acked 6732 Read 6 New Done Done false 5734 Acked 6733 Read 2 New Done Done false 5735 Acked 6734 Read 7 New Done Done false 5736 Acked 6735 Read 2 New Done Done false 5737 Acked 6736 Read 9 New Done Done false 5738 Acked 6737 Write 9 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6436 6730 6730 Downstairs last five completed: 6436 6435 6434 6433 6432 Upstairs last five completed: 5738 5737 5736 5735 5734 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:17.884991377Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:17.885110011Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:17.885180209Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:17.885206954Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:17.885227281Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:17.885249921Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:17.885501841Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:18.886378611Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:18.886446464Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:18.886474958Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:18.88650155Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:18.886522024Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:18.886544583Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:18.88680284Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1204) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.887714263Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.887951877Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.889936749Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.889969259Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.889992888Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 6436","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.890279231Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.890308857Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.890338926Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.890362212Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:19.890384382Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 6738 6737 6736 6735 6734 Upstairs last five completed: 5739 5738 5737 5736 5735 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:34:21.474 INFO request completed, latency_us: [8810] Got stop action so:false kr:false 492, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b4263f43-4456-4e16-81b5-4d6ab91757c2, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:34:21.475 INFO request completed, latency_us: 292, response_code: 200, uri: /state/cid/0, method: GET, req_id: 288278d4-83de-4b25-acd7-0253ecba54b4, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:21.476260578Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:21.476347935Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:21.476388479Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:21.476421903Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:21.476457291Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:21.476516698Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:22.477487647Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:22.47755576Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:22.477592485Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:22.477619774Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:22.477640435Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:22.477664532Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:22.477775707Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:23.478551031Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:23.478616859Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:23.478644775Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:23.478671434Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:23.478691755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:23.478735704Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:23.478795668Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:24.479906143Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:24.47997595Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:24.480015279Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:24.480042874Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:24.480063318Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:24.480085969Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:24.480132935Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:34:25.478 INFO request completed, latency_us: 243, response_code: 200, uri: /state/cid/0, method: GET, req_id: 805d68b8-ca94-408e-8c3a-497bebb4d03c, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 4566 len 20480 data: 6 6 6 6 7 002/300 Read block 2614 len 8192 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:25.481100623Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:25.481196829Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:25.481253309Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:25.481298068Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:25.481320647Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 2 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:25.481343281Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:25.481365411Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 3054 len 8192 004/300 Write block 758 len 12288 data: 4 4 3 005/300 Write block 427 len 12288 data: 4 5 6 006/300 Write block 2129 len 12288 data: 4 4 3 007/300 Read block 4483 len 12288 008/300 Read block 3628 len 40960 009/300 Read block 1598 len 12288 010/300 Read block 2046 len 24576 011/300 Write block 3674 len 4096 data: 3 012/300 Flush 013/300 Write block 2483 len 24576 data: 5 4 3 4 5 6 014/300 Read block 774 len 20480 015/300 Read block 4786 len 32768 016/300 Read block 373 len 28672 017/300 Flush 018/300 Read block 1742 len 12288 019/300 Read block 2592 len 12288 020/300 Flush 021/300 Flush 022/300 Write block 4578 len 4096 data: 7 023/300 Read block 1468 len 40960 024/300 Flush 025/300 Read block 1302 len 28672 026/300 Write block 3653 len 12288 data: 3 4 4 027/300 Read block 2423 len 20480 028/300 Read block 3113 len 12288 029/300 Read block 4290 len 28672 030/300 Write block 2127 len 4096 data: 4 031/300 Write block 1046 len 8192 data: 3 3 032/300 Write block 4887 len 24576 data: 10 9 5 4 4 4 033/300 Write block 763 len 4096 data: 2 034/300 Read block 719 len 36864 035/300 Write block 4054 len 12288 data: 3 3 2 036/300 Write block 4962 len 40960 data: 4 6 6 6 4 4 4 5 5 4 037/300 Write block 2350 len 24576 data: 3 2 2 2 2 2 038/300 Flush 039/300 Write block 4551 len 4096 data: 4 040/300 Read block 2504 len 36864 041/300 Write block 78 len 28672 data: 5 5 6 5 4 5 7 042/300 Read block 905 len 24576 043/300 Read block 4297 len 28672 044/300 Read block 450 len 28672 045/300 Write block 1139 len 20480 data: 5 4 6 5 4 046/300 Read block 2558 len 28672 047/300 Read block 1286 len 16384 048/300 Write block 1526 len 8192 data: 5 5 049/300 Read block 3422 len 12288 050/300 Write block 3991 len 4096 data: 7 051/300 Write block 2644 len 40960 data: 7 5 5 4 4 4 6 8 7 7 052/300 Flush 053/300 Read block 147 len 40960 054/300 Write block 3822 len 36864 data: 2 4 3 3 3 4 3 3 3 055/300 Write block 4250 len 24576 data: 7 6 6 7 7 7 056/300 Write block 1682 len 28672 data: 5 7 6 7 7 8 8 057/300 Write block 4233 len 20480 data: 6 7 6 6 5 058/300 Write block 1500 len 4096 data: 6 059/300 Read block 1418 len 32768 060/300 Write block 4089 len 40960 data: 5 7 7 6 4 5 5 6 4 4 061/300 Read block 1087 len 12288 062/300 Read block 4755 len 24576 063/300 Read block 4452 len 24576 064/300 Read block 4207 len 16384 065/300 Flush 066/300 Read block 723 len 4096 067/300 Read block 145 len 24576 068/300 Read block 3141 len 4096 069/300 Read block 1166 len 12288 070/300 Read block 2692 len 40960 071/300 Write block 3247 len 32768 data: 6 6 6 5 5 6 5 6 072/300 Write block 4309 len 36864 data: 5 5 3 3 3 3 3 3 4 073/300 Write block 3391 len 40960 data: 4 3 3 3 3 4 5 3 4 4 074/300 Write block 804 len 24576 data: 3 4 4 4 5 4 075/300 Read block 4874 len 12288 076/300 Read block 3131 len 8192 077/300 Write block 384 len 40960 data: 4 3 3 3 4 4 4 6 7 8 078/300 Read block 1756 len 28672 079/300 Write block 1210 len 24576 data: 4 4 4 4 4 3 080/300 Write block 3311 len 12288 data: 5 3 2 081/300 Write block 3817 len 16384 data: 3 3 3 3 082/300 Read block 3624 len 28672 083/300 Write block 34 len 32768 data: 4 4 4 3 4 5 5 5 084/300 Write block 3646 len 24576 data: 3 4 4 5 5 4 085/300 Flush 086/300 Read block 3940 len 8192 087/300 Read block 3272 len 32768 088/300 Flush 089/300 Write block 1319 len 4096 data: 3 090/300 Read block 3612 len 12288 091/300 Read block 930 len 8192 092/300 Write block 2027 len 40960 data: 2 2 3 6 7 8 8 6 6 7 093/300 Write block 3396 len 8192 data: 5 6 094/300 Flush 095/300 Write block 4929 len 32768 data: 7 6 8 7 7 7 9 8 096/300 Read block 997 len 4096 097/300 Read block 3389 len 20480 098/300 Write block 1194 len 20480 data: 2 2 2 2 2 099/300 Write block 648 len 32768 data: 4 3 3 3 3 4 4 4 100/300 Read block 4806 len 20480 101/300 Read block 1682 len 4096 102/300 Read block 4449 len 36864 103/300 Read block 474 len 16384 104/300 Read block 2756 len 16384 105/300 Write block 711 len 20480 data: 6 5 5 5 4 106/300 Write block 658 len 4096 data: 9 107/300 Read block 666 len 4096 108/300 Write block 376 len 40960 data: 5 6 6 5 5 5 5 4 5 4 109/300 Read block 2046 len 16384 110/300 Write block 2636 len 40960 data: 8 8 7 7 7 6 6 7 8 6 111/300 Write block 3841 len 4096 data: 6 112/300 Write block 1498 len 36864 data: 5 6 7 5 3 5 6 5 6 113/300 Read block 4802 len 8192 114/300 Read block 3849 len 20480 115/300 Write block 2003 len 8192 data: 4 4 116/300 Read block 2630 len 40960 117/300 Read block 3640 len 4096 118/300 Write block 1205 len 8192 data: 3 5 119/300 Read block 604 len 24576 120/300 Write block 1585 len 40960 data: 7 5 4 4 5 5 7 7 6 6 121/300 Read block 530 len 8192 122/300 Write block 2293 len 36864 data: 2 2 3 4 4 5 5 6 6 123/300 Flush 124/300 Flush 125/300 Read block 1876 len 20480 126/300 Write block 3083 len 36864 data: 7 6 5 7 4 3 3 3 3 127/300 Read block 2494 len 8192 128/300 Read block 358 len 36864 129/300 Read block 4530 len 32768 130/300 Write block 4216 len 24576 data: 7 7 8 7 8 7 131/300 Read block 3153 len 4096 132/300 Read block 541 len 24576 133/300 Read block 3786 len 12288 134/300 Write block 1197 len 40960 data: 3 3 2 3 3 3 3 4 4 6 135/300 Flush 136/300 Read block 2252 len 8192 137/300 Flush 138/300 Write block 2897 len 8192 data: 7 8 139/300 Write block 4924 len 16384 data: 6 7 8 9 140/300 Write block 3983 len 36864 data: 3 6 7 6 7 7 7 7 8 141/300 Read block 21 len 28672 142/300 Read block 3046 len 40960 143/300 Flush 144/300 Write block 2422 len 12288 data: 4 4 3 145/300 Read block 3866 len 32768 146/300 Read block 1422 len 28672 147/300 Read block 2073 len 36864 148/300 Write block 710 len 16384 data: 6 7 6 6 149/300 Read block 4235 len 12288 150/300 Read block 899 len 12288 151/300 Write block 383 len 28672 data: 5 6 5 4 4 5 5 152/300 Write block 321 len 8192 data: 6 6 153/300 Read block 2556 len 36864 154/300 Write block 1005 len 32768 data: 3 3 4 4 4 4 5 5 155/300 Write block 3504 len 40960 data: 4 4 3 3 3 3 3 3 3 6 156/300 Write block 3182 len 28672 data: 6 6 6 4 5 4 3 157/300 Flush 158/300 Write block 4375 len 24576 data: 4 4 4 4 5 5 159/300 Read block 2658 len 20480 160/300 Read block 1737 len 20480 161/300 Flush 162/300 Read block 2926 len 4096 163/300 Read block 3414 len 28672 164/300 Write block 434 len 28672 data: 6 6 8 10 11 10 10 165/300 Write block 4350 len 24576 data: 3 3 3 2 2 2 166/300 Read block 1607 len 32768 167/300 Write block 3748 len 28672 data: 3 2 2 2 2 2 2 168/300 Read block 4119 len 24576 169/300 Read block 4396 len 40960 170/300 Read block 1211 len 20480 171/300 Read block 4223 len 4096 172/300 Write block 4332 len 36864 data: 2 4 4 5 5 6 5 6 5 173/300 Write block 2995 len 24576 data: 2 2 2 2 2 2 174/300 Write block 628 len 8192 data: 3 3 175/300 Write block 3238 len 8192 data: 6 6 176/300 Flush 177/300 Write block 2829 len 32768 data: 6 6 4 3 3 3 4 4 178/300 Write block 1296 len 8192 data: 5 5 179/300 Read block 1807 len 28672 180/300 Read block 4824 len 8192 181/300 Read block 1382 len 32768 182/300 Write block 1052 len 20480 data: 4 4 4 2 2 183/300 Read block 4503 len 36864 184/300 Read block 2047 len 12288 185/300 Write block 1474 len 40960 data: 2 2 4 3 3 4 3 3 3 3 186/300 Flush 187/300 Write block 725 len 28672 data: 6 6 6 6 6 5 6 188/300 Read block 3633 len 32768 189/300 Write block 2720 len 20480 data: 4 4 3 3 3 190/300 Read block 3015 len 4096 191/300 Read block 1331 len 40960 192/300 Write block 2243 len 12288 data: 3 3 4 193/300 Write block 3959 len 28672 data: 5 5 4 4 3 3 2 194/300 Read block 2785 len 36864 195/300 Write block 3674 len 8192 data: 4 3 196/300 Read block 2244 len 16384 197/300 Write block 4441 len 16384 data: 4 6 6 7 198/300 Write block 2202 len 36864 data: 3 2 2 2 2 2 3 5 5 199/300 Write block 4778 len 24576 data: 2 2 2 2 3 3 200/300 Write block 2389 len 24576 data: 6 6 7 5 5 4 201/300 Read block 3044 len 12288 202/300 Read block 627 len 20480 203/300 Read block 3784 len 36864 204/300 Read block 301 len 40960 205/300 Write block 2539 len 36864 data: 4 5 4 4 4 5 5 6 4 206/300 Read block 1672 len 36864 207/300 Read block 3159 len 12288 208/300 Flush 209/300 Write block 4287 len 12288 data: 4 3 3 210/300 Write block 3302 len 40960 data: 4 4 5 5 5 4 3 4 5 6 211/300 Read block 74 len 8192 212/300 Write block 4502 len 36864 data: 5 7 7 6 6 5 5 5 4 213/300 Write block 4694 len 16384 data: 4 7 7 7 214/300 Read block 691 len 36864 215/300 Write block 95 len 12288 data: 5 5 5 216/300 Write block 1716 len 32768 data: 4 4 4 4 3 3 3 3 217/300 Read block 2837 len 28672 218/300 Write block 1066 len 40960 data: 3 3 3 4 7 6 8 8 8 7 219/300 Write block 267 len 20480 data: 8 9 8 8 9 220/300 Write block 3892 len 32768 data: 4 4 4 5 4 5 5 4 221/300 Write block 2944 len 12288 data: 3 4 4 222/300 Flush 223/300 Write block 2821 len 36864 data: 9 10 11 9 10 10 9 8 7 224/300 Write block 2423 len 12288 data: 5 4 2 225/300 Read block 3358 len 24576 226/300 Read block 1424 len 20480 227/300 Write block 124 len 40960 data: 4 5 4 3 4 5 5 5 4 5 228/300 Read block 693 len 28672 229/300 Read block 85 len 20480 230/300 Read block 3943 len 8192 231/300 Read block 1949 len 12288 232/300 Write block 3213 len 32768 data: 4 4 4 2 2 2 2 3 233/300 Flush 234/300 Write block 2553 len 20480 data: 3 3 4 3 3 235/300 Write block 221 len 16384 data: 4 5 5 5 236/300 Flush 237/300 Write block 1296 len 32768 data: 6 6 6 5 6 5 5 4 238/300 Read block 3700 len 8192 239/300 Flush 240/300 Read block 1749 len 16384 241/300 Write block 2329 len 20480 data: 8 6 6 6 4 242/300 Write block 1665 len 36864 data: 4 4 4 5 7 6 6 5 5 243/300 Write block 3959 len 36864 data: 6 6 5 5 4 4 3 2 2 244/300 Read block 4878 len 36864 245/300 Read block 3046 len 32768 246/300 Write block 3092 len 32768 data: 3 3 3 4 3 3 4 5 247/300 Read block 3317 len 8192 248/300 Read block 3157 len 20480 249/300 Read block 3991 len 24576 250/300 Write block 3175 len 16384 data: 3 3 4 7 251/300 Read block 3241 len 4096 252/300 Write block 1513 len 28672 data: 3 2 2 2 2 3 4 253/300 Write block 4007 len 12288 data: 6 7 9 254/300 Flush 255/300 Write block 227 len 24576 data: 4 3 2 2 3 3 256/300 Write block 484 len 8192 data: 4 4 257/300 Flush 258/300 Read block 1505 len 8192 259/300 Read block 2528 len 16384 260/300 Read block 575 len 12288 261/300 Write block 1605 len 32768 data: 3 3 2 3 3 4 4 4 262/300 Write block 1162 len 36864 data: 6 6 5 5 6 4 4 5 4 263/300 Read block 2766 len 4096 264/300 Read block 2113 len 36864 265/300 Write block 1545 len 4096 data: 3 266/300 Flush 267/300 Read block 4891 len 8192 268/300 Read block 1778 len 8192 269/300 Write block 799 len 20480 data: 3 3 5 5 5 270/300 Read block 2283 len 32768 271/300 Write block 3229 len 24576 data: 7 7 6 6 6 6 272/300 Read block 3568 len 12288 273/300 Write block 35 len 8192 data: 5 5 274/300 Read block 3851 len 36864 275/300 Write block 1089 len 28672 data: 5 5 5 5 5 6 5 276/300 Read block 3591 len 4096 277/300 Write block 1454 len 32768 data: 3 3 2 3 3 3 4 5 278/300 Write block 2818 len 28672 data: 5 6 7 10 11 12 10 279/300 Read block 337 len 20480 280/300 Read block 4305 len 36864 281/300 Write block 4924 len 40960 data: 7 8 9 10 9 8 7 9 8 8 282/300 Read block 2940 len 12288 283/300 Write block 165 len 16384 data: 4 4 4 4 284/300 Read block 4872 len 40960 285/300 Write block 3320 len 16384 data: 6 5 3 3 286/300 Write block 584 len 24576 data: 5 5 4 4 4 4 287/300 Read block 2404 len 8192 288/300 Read block 4372 len 36864 289/300 Write block 2285 len 24576 data: 4 4 4 3 3 2 290/300 Read block 2590 len 20480 291/300 Read block 4077 len 4096 292/300 Write block 2842 len 12288 data: 5 7 7 293/300 Write block 2835 len 36864 data: 5 5 5 7 7 7 5 6 8 294/300 Write block 366 len 20480 data: 8 9 6 4 3 295/300 Write block 3063 len 8192 data: 6 6 296/300 Read block 1698 len 8192 297/300 Read block 1788 len 12288 298/300 Read block 3785 len 4096 299/300 Flush 300/300 Write block 211 len 4096 data: 5 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:34:26.046 INFO request completed, latency_us: 287, response_code: 204, uri: /start/cid/0, method: POST, req_id: 2c102edd-20a2-474b-8c3b-3360630b66d9, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 5740 Acked 6739 Write 5 New Done Done true 5741 Acked 6740 Read 2 New Done Done true 5742 Acked 6741 Read 2 New Done Done false 5743 Acked 6742 Write 3 New Done Done false 5744 Acked 6743 Write 3 New Done Done false 5745 Acked 6744 Write 3 New Done Done false 5746 Acked 6745 Read 3 New Done Done false 5747 Acked 6746 Read 10 New Done Done false 5748 Acked 6747 Read 3 New Done Done false 5749 Acked 6748 Read 6 New Done Done false 5750 Acked 6749 Write 1 New Done Done false 5751 Acked 6750 Flush 0 New Done Done false 5752 Acked 6751 Write 6 New Done Done false 5753 Acked 6752 Read 5 New Done Done false 5754 Acked 6753 Read 8 New Done Done false 5755 Acked 6754 Read 7 New Done Done false 5756 Acked 6755 Flush 0 New Done Done false 5757 Acked 6756 Read 3 New Done Done false 5758 Acked 6757 Read 3 New Done Done false 5759 Acked 6758 Flush 0 New Done Done false 5760 Acked 6759 Flush 0 New Done Done false 5761 Acked 6760 Write 1 New Done Done false 5762 Acked 6761 Read 10 New Done Done false 5763 Acked 6762 Flush 0 New Done Done false 5764 Acked 6763 Read 7 New Done Done false 5765 Acked 6764 Write 3 New Done Done false 5766 Acked 6765 Read 5 New Done Done false 5767 Acked 6766 Read 3 New Done Done false 5768 Acked 6767 Read 7 New Done Done false 5769 Acked 6768 Write 1 New Done Done false 5770 Acked 6769 Write 2 New Done Done false 5771 Acked 6770 Write 6 New Done Done false 5772 Acked 6771 Write 1 New Done Done false 5773 Acked 6772 Read 9 New Done Done false 5774 Acked 6773 Write 3 New Done Done false 5775 Acked 6774 Write 10 New Done Done false 5776 Acked 6775 Write 6 New Done Done false 5777 Acked 6776 Flush 0 New Done Done false 5778 Acked 6777 Write 1 New Done Done false 5779 Acked 6778 Read 9 New Done Done false 5780 Acked 6779 Write 7 New Done Done false 5781 Acked 6780 Read 6 New Done Done false 5782 Acked 6781 Read 7 New Done Done false 5783 Acked 6782 Read 7 New Done Done false 5784 Acked 6783 Write 5 New Done Done false 5785 Acked 6784 Read 7 New Done Done false 5786 Acked 6785 Read 4 New Done Done false 5787 Acked 6786 Write 2 New Done Done false 5788 Acked 6787 Read 3 New Done Done false 5789 Acked 6788 Write 1 New Done Done false 5790 Acked 6789 Write 10 New Done Done false 5791 Acked 6790 Flush 0 New Done Done false 5792 Acked 6791 Read 10 New Done Done false 5793 Acked 6792 Write 9 New Done Done false 5794 Acked 6793 Write 6 New Done Done false 5795 Acked 6794 Write 7 New Done Done false 5796 Acked 6795 Write 5 New Done Done false 5797 Acked 6796 Write 1 New Done Done false 5798 Acked 6797 Read 8 New Done Done false 5799 Acked 6798 Write 10 New Done Done false 5800 Acked 6799 Read 3 New Done Done false 5801 Acked 6800 Read 6 New Done Done false 5802 Acked 6801 Read 6 New Done Done false 5803 Acked 6802 Read 4 New Done Done false 5804 Acked 6803 Flush 0 New Done Done false 5805 Acked 6804 Read 1 New Done Done false 5806 Acked 6805 Read 6 New Done Done false 5807 Acked 6806 Read 1 New Done Done false 5808 Acked 6807 Read 3 New Done Done false 5809 Acked 6808 Read 10 New Done Done false 5810 Acked 6809 Write 8 New Done Done false 5811 Acked 6810 Write 9 New Done Done false 5812 Acked 6811 Write 10 New Done Done false 5813 Acked 6812 Write 6 New Done Done false 5814 Acked 6813 Read 3 New Done Done false 5815 Acked 6814 Read 2 New Done Done false 5816 Acked 6815 Write 10 New Done Done false 5817 Acked 6816 Read 7 New Done Done false 5818 Acked 6817 Write 6 New Done Done false 5819 Acked 6818 Write 3 New Done Done false 5820 Acked 6819 Write 4 New Done Done false 5821 Acked 6820 Read 7 New Done Done false 5822 Acked 6821 Write 8 New Done Done false 5823 Acked 6822 Write 6 New Done Done false 5824 Acked 6823 Flush 0 New Done Done false 5825 Acked 6824 Read 2 New Done Done false 5826 Acked 6825 Read 8 New Done Done false 5827 Acked 6826 Flush 0 New Done Done false 5828 Acked 6827 Write 1 New Done Done false 5829 Acked 6828 Read 3 New Done Done false 5830 Acked 6829 Read 2 New Done Done false 5831 Acked 6830 Write 10 New Done Done false 5832 Acked 6831 Write 2 New Done Done false 5833 Acked 6832 Flush 0 New Done Done false 5834 Acked 6833 Write 8 New Done Done false 5835 Acked 6834 Read 1 New Done Done false 5836 Acked 6835 Read 5 New Done Done false 5837 Acked 6836 Write 5 New Done Done false 5838 Acked 6837 Write 8 New Done Done false 5839 Acked 6838 Read 5 New Done Done false 5840 Acked 6839 Read 1 New Done Done false 5841 Acked 6840 Read 9 New Done Done false 5842 Acked 6841 Read 4 New Done Done false 5843 Acked 6842 Read 4 New Done Done false 5844 Acked 6843 Write 5 New Done Done false 5845 Acked 6844 Write 1 New Done Done false 5846 Acked 6845 Read 1 New Done Done false 5847 Acked 6846 Write 10 New Done Done false 5848 Acked 6847 Read 4 New Done Done false 5849 Acked 6848 Write 10 New Done Done false 5850 Acked 6849 Write 1 New Done Done false 5851 Acked 6850 Write 9 New Done Done false 5852 Acked 6851 Read 2 New Done Done false 5853 Acked 6852 Read 5 New Done Done false 5854 Acked 6853 Write 2 New Done Done false 5855 Acked 6854 Read 10 New Done Done false 5856 Acked 6855 Read 1 New Done Done false 5857 Acked 6856 Write 2 New Done Done false 5858 Acked 6857 Read 6 New Done Done false 5859 Acked 6858 Write 10 New Done Done false 5860 Acked 6859 Read 2 New Done Done false 5861 Acked 6860 Write 9 New Done Done false 5862 Acked 6861 Flush 0 New Done Done false 5863 Acked 6862 Flush 0 New Done Done false 5864 Acked 6863 Read 5 New Done Done false 5865 Acked 6864 Write 9 New Done Done false 5866 Acked 6865 Read 2 New Done Done false 5867 Acked 6866 Read 9 New Done Done false 5868 Acked 6867 Read 8 New Done Done false 5869 Acked 6868 Write 6 New Done Done false 5870 Acked 6869 Read 1 New Done Done false 5871 Acked 6870 Read 6 New Done Done false 5872 Acked 6871 Read 3 New Done Done false 5873 Acked 6872 Write 10 New Done Done false 5874 Acked 6873 Flush 0 New Done Done false 5875 Acked 6874 Read 2 New Done Done false 5876 Acked 6875 Flush 0 New Done Done false 5877 Acked 6876 Write 2 New Done Done false 5878 Acked 6877 Write 4 New Done Done false 5879 Acked 6878 Write 9 New Done Done false 5880 Acked 6879 Read 7 New Done Done false 5881 Acked 6880 Read 10 New Done Done false 5882 Acked 6881 Flush 0 New Done Done false 5883 Acked 6882 Write 3 New Done Done false 5884 Acked 6883 Read 8 New Done Done false 5885 Acked 6884 Read 7 New Done Done false 5886 Acked 6885 Read 9 New Done Done false 5887 Acked 6886 Write 4 New Done Done false 5888 Acked 6887 Read 3 New Done Done false 5889 Acked 6888 Read 3 New Done Done false 5890 Acked 6889 Write 7 New Done Done false 5891 Acked 6890 Write 2 New Done Done false 5892 Acked 6891 Read 9 New Done Done false 5893 Acked 6892 Write 8 New Done Done false 5894 Acked 6893 Write 10 New Done Done false 5895 Acked 6894 Write 7 New Done Done false 5896 Acked 6895 Flush 0 New Done Done false 5897 Acked 6896 Write 6 New Done Done false 5898 Acked 6897 Read 5 New Done Done false 5899 Acked 6898 Read 5 New Done Done false 5900 Acked 6899 Flush 0 New Done Done false 5901 Acked 6900 Read 1 New Done Done false 5902 Acked 6901 Read 7 New Done Done false 5903 Acked 6902 Write 7 New Done Done false 5904 Acked 6903 Write 6 New Done Done false 5905 Acked 6904 Read 8 New Done Done false 5906 Acked 6905 Write 7 New Done Done false 5907 Acked 6906 Read 6 New Done Done false 5908 Acked 6907 Read 10 New Done Done false 5909 Acked 6908 Read 5 New Done Done false 5910 Acked 6909 Read 1 New Done Done false 5911 Acked 6910 Write 9 New Done Done false 5912 Acked 6911 Write 6 New Done Done false 5913 Acked 6912 Write 2 New Done Done false 5914 Acked 6913 Write 2 New Done Done false 5915 Acked 6914 Flush 0 New Done Done false 5916 Acked 6915 Write 8 New Done Done false 5917 Acked 6916 Write 2 New Done Done false 5918 Acked 6917 Read 7 New Done Done false 5919 Acked 6918 Read 2 New Done Done false 5920 Acked 6919 Read 8 New Done Done false 5921 Acked 6920 Write 5 New Done Done false 5922 Acked 6921 Read 9 New Done Done false 5923 Acked 6922 Read 3 New Done Done false 5924 Acked 6923 Write 10 New Done Done false 5925 Acked 6924 Flush 0 New Done Done false 5926 Acked 6925 Write 7 New Done Done false 5927 Acked 6926 Read 8 New Done Done false 5928 Acked 6927 Write 5 New Done Done false 5929 Acked 6928 Read 1 New Done Done false 5930 Acked 6929 Read 10 New Done Done false 5931 Acked 6930 Write 3 New Done Done false 5932 Acked 6931 Write 7 New Done Done false 5933 Acked 6932 Read 9 New Done Done false 5934 Acked 6933 Write 2 New Done Done false 5935 Acked 6934 Read 4 New Done Done false 5936 Acked 6935 Write 4 New Done Done false 5937 Acked 6936 Write 9 New Done Done false 5938 Acked 6937 Write 6 New Done Done false 5939 Acked 6938 Write 6 New Done Done false 5940 Acked 6939 Read 3 New Done Done false 5941 Acked 6940 Read 5 New Done Done false 5942 Acked 6941 Read 9 New Done Done false 5943 Acked 6942 Read 10 New Done Done false 5944 Acked 6943 Write 9 New Done Done false 5945 Acked 6944 Read 9 New Done Done false 5946 Acked 6945 Read 3 New Done Done false 5947 Acked 6946 Flush 0 New Done Done false 5948 Acked 6947 Write 3 New Done Done false 5949 Acked 6948 Write 10 New Done Done false 5950 Acked 6949 Read 2 New Done Done false 5951 Acked 6950 Write 9 New Done Done false 5952 Acked 6951 Write 4 New Done Done false 5953 Acked 6952 Read 9 New Done Done false 5954 Acked 6953 Write 3 New Done Done false 5955 Acked 6954 Write 8 New Done Done false 5956 Acked 6955 Read 7 New Done Done false 5957 Acked 6956 Write 10 New Done Done false 5958 Acked 6957 Write 5 New Done Done false 5959 Acked 6958 Write 8 New Done Done false 5960 Acked 6959 Write 3 New Done Done false 5961 Acked 6960 Flush 0 New Done Done false 5962 Acked 6961 Write 9 New Done Done false 5963 Acked 6962 Write 3 New Done Done false 5964 Acked 6963 Read 6 New Done Done false 5965 Acked 6964 Read 5 New Done Done false 5966 Acked 6965 Write 10 New Done Done false 5967 Acked 6966 Read 7 New Done Done false 5968 Acked 6967 Read 5 New Done Done false 5969 Acked 6968 Read 2 New Done Done false 5970 Acked 6969 Read 3 New Done Done false 5971 Acked 6970 Write 8 New Done Done false 5972 Acked 6971 Flush 0 New Done Done false 5973 Acked 6972 Write 5 New Done Done false 5974 Acked 6973 Write 4 New Done Done false 5975 Acked 6974 Flush 0 New Done Done false 5976 Acked 6975 Write 8 New Done Done false 5977 Acked 6976 Read 2 New Done Done false 5978 Acked 6977 Flush 0 New Done Done false 5979 Acked 6978 Read 4 New Done Done false 5980 Acked 6979 Write 5 New Done Done false 5981 Acked 6980 Write 9 New Done Done false 5982 Acked 6981 Write 9 New Done Done false 5983 Acked 6982 Read 9 New Done Done false 5984 Acked 6983 Read 8 New Done Done false 5985 Acked 6984 Write 8 New Done Done false 5986 Acked 6985 Read 2 New Done Done false 5987 Acked 6986 Read 5 New Done Done false 5988 Acked 6987 Read 6 New Done Done false 5989 Acked 6988 Write 4 New Done Done false 5990 Acked 6989 Read 1 New Done Done false 5991 Acked 6990 Write 7 New Done Done false 5992 Acked 6991 Write 3 New Done Done false 5993 Acked 6992 Flush 0 New Done Done false 5994 Acked 6993 Write 6 New Done Done false 5995 Acked 6994 Write 2 New Done Done false 5996 Acked 6995 Flush 0 New Done Done false 5997 Acked 6996 Read 2 New Done Done false 5998 Acked 6997 Read 4 New Done Done false 5999 Acked 6998 Read 3 New Done Done false 6000 Acked 6999 Write 8 New Done Done false 6001 Acked 7000 Write 9 New Done Done false 6002 Acked 7001 Read 1 New Done Done false 6003 Acked 7002 Read 9 New Done Done false 6004 Acked 7003 Write 1 New Done Done false 6005 Acked 7004 Flush 0 New Done Done false 6006 Acked 7005 Read 2 New Done Done false 6007 Acked 7006 Read 2 New Done Done false 6008 Acked 7007 Write 5 New Done Done false 6009 Acked 7008 Read 8 New Done Done false 6010 Acked 7009 Write 6 New Done Done false 6011 Acked 7010 Read 3 New Done Done false 6012 Acked 7011 Write 2 New Done Done false 6013 Acked 7012 Read 9 New Done Done false 6014 Acked 7013 Write 7 New Done Done false 6015 Acked 7014 Read 1 New Done Done false 6016 Acked 7015 Write 8 New Done Done false 6017 Acked 7016 Write 7 New Done Done false 6018 Acked 7017 Read 5 New Done Done false 6019 Acked 7018 Read 9 New Done Done false 6020 Acked 7019 Write 10 New Done Done false 6021 Acked 7020 Read 3 New Done Done false 6022 Acked 7021 Write 4 New Done Done false 6023 Acked 7022 Read 10 New Done Done false 6024 Acked 7023 Write 4 New Done Done false 6025 Acked 7024 Write 6 New Done Done false 6026 Acked 7025 Read 2 New Done Done false 6027 Acked 7026 Read 9 New Done Done false 6028 Acked 7027 Write 6 New Done Done false 6029 Acked 7028 Read 5 New Done Done false 6030 Acked 7029 Read 1 New Done Done false 6031 Acked 7030 Write 3 New Done Done false 6032 Acked 7031 Write 9 New Done Done false 6033 Acked 7032 Write 5 New Done Done false 6034 Acked 7033 Write 2 New Done Done false 6035 Acked 7034 Read 2 New Done Done false 6036 Acked 7035 Read 3 New Done Done false 6037 Acked 7036 Read 1 New Done Done false 6038 Acked 7037 Flush 0 New Done Done false 6039 Acked 7038 Write 1 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 300 0 0 300 Sent 0 0 0 0 Done 0 300 300 600 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6738 7037 7037 Downstairs last five completed: 6738 6737 6736 6735 6734 Upstairs last five completed: 6039 6038 6037 6036 6035 CLIENT: Up:0 ds:300 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:26.482641248Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:26.482708365Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:26.482736865Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:26.482761247Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:26.482787569Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:26.482807799Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:26.48283035Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:26.483047936Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:27.483949159Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:27.484017667Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:27.484046062Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:27.484083061Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:27.48410449Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:27.484136625Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:27.484389129Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:28.48520644Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:28.485273553Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:28.485302057Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:28.485328745Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:28.485349076Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:28.485383545Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:28.485638468Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:29.486396681Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:29.48646757Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:29.486506885Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:29.486534374Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:29.486554641Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:29.486577197Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:29.48686542Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 5740 Acked 6739 Write 5 New Done Done true 5741 Acked 6740 Read 2 New Done Done true 5742 Acked 6741 Read 2 New Done Done true 5743 Acked 6742 Write 3 New Done Done true 5744 Acked 6743 Write 3 New Done Done true 5745 Acked 6744 Write 3 New Done Done true 5746 Acked 6745 Read 3 New Done Done true 5747 Acked 6746 Read 10 New Done Done true 5748 Acked 6747 Read 3 New Done Done true 5749 Acked 6748 Read 6 New Done Done true 5750 Acked 6749 Write 1 New Done Done true 5751 Acked 6750 Flush 0 New Done Done true 5752 Acked 6751 Write 6 New Done Done true 5753 Acked 6752 Read 5 New Done Done true 5754 Acked 6753 Read 8 New Done Done true 5755 Acked 6754 Read 7 New Done Done true 5756 Acked 6755 Flush 0 New Done Done true 5757 Acked 6756 Read 3 New Done Done true 5758 Acked 6757 Read 3 New Done Done true 5759 Acked 6758 Flush 0 New Done Done true 5760 Acked 6759 Flush 0 New Done Done true 5761 Acked 6760 Write 1 New Done Done true 5762 Acked 6761 Read 10 New Done Done true 5763 Acked 6762 Flush 0 New Done Done true 5764 Acked 6763 Read 7 New Done Done true 5765 Acked 6764 Write 3 New Done Done true 5766 Acked 6765 Read 5 New Done Done true 5767 Acked 6766 Read 3 New Done Done true 5768 Acked 6767 Read 7 New Done Done true 5769 Acked 6768 Write 1 New Done Done true 5770 Acked 6769 Write 2 New Done Done true 5771 Acked 6770 Write 6 New Done Done true 5772 Acked 6771 Write 1 New Done Done true 5773 Acked 6772 Read 9 New Done Done true 5774 Acked 6773 Write 3 New Done Done true 5775 Acked 6774 Write 10 New Done Done true 5776 Acked 6775 Write 6 New Done Done true 5777 Acked 6776 Flush 0 New Done Done true 5778 Acked 6777 Write 1 New Done Done true 5779 Acked 6778 Read 9 New Done Done true 5780 Acked 6779 Write 7 New Done Done true 5781 Acked 6780 Read 6 New Done Done true 5782 Acked 6781 Read 7 New Done Done true 5783 Acked 6782 Read 7 New Done Done true 5784 Acked 6783 Write 5 New Done Done true 5785 Acked 6784 Read 7 New Done Done true 5786 Acked 6785 Read 4 New Done Done true 5787 Acked 6786 Write 2 New Done Done true 5788 Acked 6787 Read 3 New Done Done true 5789 Acked 6788 Write 1 New Done Done true 5790 Acked 6789 Write 10 New Done Done true 5791 Acked 6790 Flush 0 New Done Done true 5792 Acked 6791 Read 10 New Done Done true 5793 Acked 6792 Write 9 New Done Done true 5794 Acked 6793 Write 6 New Done Done true 5795 Acked 6794 Write 7 New Done Done true 5796 Acked 6795 Write 5 New Done Done true 5797 Acked 6796 Write 1 New Done Done true 5798 Acked 6797 Read 8 New Done Done true 5799 Acked 6798 Write 10 New Done Done true 5800 Acked 6799 Read 3 New Done Done true 5801 Acked 6800 Read 6 New Done Done true 5802 Acked 6801 Read 6 New Done Done true 5803 Acked 6802 Read 4 New Done Done true 5804 Acked 6803 Flush 0 New Done Done true 5805 Acked 6804 Read 1 New Done Done true 5806 Acked 6805 Read 6 New Done Done true 5807 Acked 6806 Read 1 New Done Done true 5808 Acked 6807 Read 3 New Done Done true 5809 Acked 6808 Read 10 New Done Done true 5810 Acked 6809 Write 8 New Done Done true 5811 Acked 6810 Write 9 New Done Done true 5812 Acked 6811 Write 10 New Done Done true 5813 Acked 6812 Write 6 New Done Done true 5814 Acked 6813 Read 3 New Done Done true 5815 Acked 6814 Read 2 New Done Done true 5816 Acked 6815 Write 10 New Done Done true 5817 Acked 6816 Read 7 New Done Done true 5818 Acked 6817 Write 6 New Done Done true 5819 Acked 6818 Write 3 New Done Done true 5820 Acked 6819 Write 4 New Done Done true 5821 Acked 6820 Read 7 New Done Done true 5822 Acked 6821 Write 8 New Done Done true 5823 Acked 6822 Write 6 New Done Done true 5824 Acked 6823 Flush 0 New Done Done true 5825 Acked 6824 Read 2 New Done Done true 5826 Acked 6825 Read 8 New Done Done true 5827 Acked 6826 Flush 0 New Done Done true 5828 Acked 6827 Write 1 New Done Done true 5829 Acked 6828 Read 3 New Done Done true 5830 Acked 6829 Read 2 New Done Done true 5831 Acked 6830 Write 10 New Done Done true 5832 Acked 6831 Write 2 New Done Done true 5833 Acked 6832 Flush 0 New Done Done true 5834 Acked 6833 Write 8 New Done Done true 5835 Acked 6834 Read 1 New Done Done true 5836 Acked 6835 Read 5 New Done Done true 5837 Acked 6836 Write 5 New Done Done true 5838 Acked 6837 Write 8 New Done Done true 5839 Acked 6838 Read 5 New Done Done true 5840 Acked 6839 Read 1 New Done Done true 5841 Acked 6840 Read 9 New Done Done true 5842 Acked 6841 Read 4 New Done Done true 5843 Acked 6842 Read 4 New Done Done true 5844 Acked 6843 Write 5 New Done Done true 5845 Acked 6844 Write 1 New Done Done true 5846 Acked 6845 Read 1 New Done Done true 5847 Acked 6846 Write 10 New Done Done true 5848 Acked 6847 Read 4 New Done Done true 5849 Acked 6848 Write 10 New Done Done true 5850 Acked 6849 Write 1 New Done Done true 5851 Acked 6850 Write 9 New Done Done true 5852 Acked 6851 Read 2 New Done Done true 5853 Acked 6852 Read 5 New Done Done true 5854 Acked 6853 Write 2 New Done Done true 5855 Acked 6854 Read 10 New Done Done true 5856 Acked 6855 Read 1 New Done Done true 5857 Acked 6856 Write 2 New Done Done true 5858 Acked 6857 Read 6 New Done Done true 5859 Acked 6858 Write 10 New Done Done true 5860 Acked 6859 Read 2 New Done Done true 5861 Acked 6860 Write 9 New Done Done true 5862 Acked 6861 Flush 0 New Done Done true 5863 Acked 6862 Flush 0 New Done Done true 5864 Acked 6863 Read 5 New Done Done true 5865 Acked 6864 Write 9 New Done Done true 5866 Acked 6865 Read 2 New Done Done true 5867 Acked 6866 Read 9 New Done Done true 5868 Acked 6867 Read 8 New Done Done true 5869 Acked 6868 Write 6 New Done Done true 5870 Acked 6869 Read 1 New Done Done true 5871 Acked 6870 Read 6 New Done Done true 5872 Acked 6871 Read 3 New Done Done true 5873 Acked 6872 Write 10 New Done Done true 5874 Acked 6873 Flush 0 New Done Done true 5875 Acked 6874 Read 2 New Done Done true 5876 Acked 6875 Flush 0 New Done Done true 5877 Acked 6876 Write 2 New Done Done true 5878 Acked 6877 Write 4 New Done Done true 5879 Acked 6878 Write 9 New Done Done true 5880 Acked 6879 Read 7 New Done Done true 5881 Acked 6880 Read 10 New Done Done true 5882 Acked 6881 Flush 0 New Done Done true 5883 Acked 6882 Write 3 New Done Done true 5884 Acked 6883 Read 8 New Done Done true 5885 Acked 6884 Read 7 New Done Done true 5886 Acked 6885 Read 9 New Done Done true 5887 Acked 6886 Write 4 New Done Done true 5888 Acked 6887 Read 3 New Done Done true 5889 Acked 6888 Read 3 New Done Done true 5890 Acked 6889 Write 7 New Done Done true 5891 Acked 6890 Write 2 New Done Done true 5892 Acked 6891 Read 9 New Done Done true 5893 Acked 6892 Write 8 New Done Done true 5894 Acked 6893 Write 10 New Done Done true 5895 Acked 6894 Write 7 New Done Done true 5896 Acked 6895 Flush 0 New Done Done true 5897 Acked 6896 Write 6 New Done Done true 5898 Acked 6897 Read 5 New Done Done true 5899 Acked 6898 Read 5 New Done Done true 5900 Acked 6899 Flush 0 New Done Done true 5901 Acked 6900 Read 1 New Done Done true 5902 Acked 6901 Read 7 New Done Done true 5903 Acked 6902 Write 7 New Done Done true 5904 Acked 6903 Write 6 New Done Done true 5905 Acked 6904 Read 8 New Done Done true 5906 Acked 6905 Write 7 New Done Done true 5907 Acked 6906 Read 6 New Done Done true 5908 Acked 6907 Read 10 New Done Done true 5909 Acked 6908 Read 5 New Done Done true 5910 Acked 6909 Read 1 New Done Done true 5911 Acked 6910 Write 9 New Done Done true 5912 Acked 6911 Write 6 New Done Done true 5913 Acked 6912 Write 2 New Done Done true 5914 Acked 6913 Write 2 New Done Done true 5915 Acked 6914 Flush 0 New Done Done true 5916 Acked 6915 Write 8 New Done Done true 5917 Acked 6916 Write 2 New Done Done true 5918 Acked 6917 Read 7 New Done Done true 5919 Acked 6918 Read 2 New Done Done true 5920 Acked 6919 Read 8 New Done Done true 5921 Acked 6920 Write 5 New Done Done true 5922 Acked 6921 Read 9 New Done Done true 5923 Acked 6922 Read 3 New Done Done true 5924 Acked 6923 Write 10 New Done Done true 5925 Acked 6924 Flush 0 New Done Done true 5926 Acked 6925 Write 7 New Done Done true 5927 Acked 6926 Read 8 New Done Done true 5928 Acked 6927 Write 5 New Done Done true 5929 Acked 6928 Read 1 New Done Done true 5930 Acked 6929 Read 10 New Done Done true 5931 Acked 6930 Write 3 New Done Done true 5932 Acked 6931 Write 7 New Done Done true 5933 Acked 6932 Read 9 New Done Done true 5934 Acked 6933 Write 2 New Done Done true 5935 Acked 6934 Read 4 New Done Done true 5936 Acked 6935 Write 4 New Done Done true 5937 Acked 6936 Write 9 New Done Done true 5938 Acked 6937 Write 6 New Done Done true 5939 Acked 6938 Write 6 New Done Done true 5940 Acked 6939 Read 3 New Done Done true 5941 Acked 6940 Read 5 New Done Done true 5942 Acked 6941 Read 9 New Done Done true 5943 Acked 6942 Read 10 New Done Done true 5944 Acked 6943 Write 9 New Done Done true 5945 Acked 6944 Read 9 New Done Done true 5946 Acked 6945 Read 3 New Done Done true 5947 Acked 6946 Flush 0 New Done Done true 5948 Acked 6947 Write 3 New Done Done true 5949 Acked 6948 Write 10 New Done Done true 5950 Acked 6949 Read 2 New Done Done true 5951 Acked 6950 Write 9 New Done Done true 5952 Acked 6951 Write 4 New Done Done true 5953 Acked 6952 Read 9 New Done Done true 5954 Acked 6953 Write 3 New Done Done true 5955 Acked 6954 Write 8 New Done Done true 5956 Acked 6955 Read 7 New Done Done true 5957 Acked 6956 Write 10 New Done Done true 5958 Acked 6957 Write 5 New Done Done true 5959 Acked 6958 Write 8 New Done Done true 5960 Acked 6959 Write 3 New Done Done true 5961 Acked 6960 Flush 0 New Done Done true 5962 Acked 6961 Write 9 New Done Done true 5963 Acked 6962 Write 3 New Done Done true 5964 Acked 6963 Read 6 New Done Done true 5965 Acked 6964 Read 5 New Done Done true 5966 Acked 6965 Write 10 New Done Done true 5967 Acked 6966 Read 7 New Done Done true 5968 Acked 6967 Read 5 New Done Done true 5969 Acked 6968 Read 2 New Done Done true 5970 Acked 6969 Read 3 New Done Done true 5971 Acked 6970 Write 8 New Done Done true 5972 Acked 6971 Flush 0 New Done Done true 5973 Acked 6972 Write 5 New Done Done true 5974 Acked 6973 Write 4 New Done Done true 5975 Acked 6974 Flush 0 New Done Done true 5976 Acked 6975 Write 8 New Done Done true 5977 Acked 6976 Read 2 New Done Done true 5978 Acked 6977 Flush 0 New Done Done true 5979 Acked 6978 Read 4 New Done Done true 5980 Acked 6979 Write 5 New Done Done true 5981 Acked 6980 Write 9 New Done Done true 5982 Acked 6981 Write 9 New Done Done true 5983 Acked 6982 Read 9 New Done Done true 5984 Acked 6983 Read 8 New Done Done true 5985 Acked 6984 Write 8 New Done Done true 5986 Acked 6985 Read 2 New Done Done true 5987 Acked 6986 Read 5 New Done Done true 5988 Acked 6987 Read 6 New Done Done true 5989 Acked 6988 Write 4 New Done Done true 5990 Acked 6989 Read 1 New Done Done true 5991 Acked 6990 Write 7 New Done Done true 5992 Acked 6991 Write 3 New Done Done true 5993 Acked 6992 Flush 0 New Done Done true 5994 Acked 6993 Write 6 New Done Done true 5995 Acked 6994 Write 2 New Done Done true 5996 Acked 6995 Flush 0 New Done Done true 5997 Acked 6996 Read 2 New Done Done true 5998 Acked 6997 Read 4 New Done Done true 5999 Acked 6998 Read 3 New Done Done true 6000 Acked 6999 Write 8 New Done Done true 6001 Acked 7000 Write 9 New Done Done true 6002 Acked 7001 Read 1 New Done Done true 6003 Acked 7002 Read 9 New Done Done true 6004 Acked 7003 Write 1 New Done Done true 6005 Acked 7004 Flush 0 New Done Done true 6006 Acked 7005 Read 2 New Done Done true 6007 Acked 7006 Read 2 New Done Done true 6008 Acked 7007 Write 5 New Done Done true 6009 Acked 7008 Read 8 New Done Done true 6010 Acked 7009 Write 6 New Done Done true 6011 Acked 7010 Read 3 New Done Done true 6012 Acked 7011 Write 2 New Done Done true 6013 Acked 7012 Read 9 New Done Done true 6014 Acked 7013 Write 7 New Done Done true 6015 Acked 7014 Read 1 New Done Done true 6016 Acked 7015 Write 8 New Done Done true 6017 Acked 7016 Write 7 New Done Done true 6018 Acked 7017 Read 5 New Done Done true 6019 Acked 7018 Read 9 New Done Done true 6020 Acked 7019 Write 10 New Done Done true 6021 Acked 7020 Read 3 New Done Done true 6022 Acked 7021 Write 4 New Done Done true 6023 Acked 7022 Read 10 New Done Done true 6024 Acked 7023 Write 4 New Done Done true 6025 Acked 7024 Write 6 New Done Done true 6026 Acked 7025 Read 2 New Done Done true 6027 Acked 7026 Read 9 New Done Done true 6028 Acked 7027 Write 6 New Done Done true 6029 Acked 7028 Read 5 New Done Done true 6030 Acked 7029 Read 1 New Done Done true 6031 Acked 7030 Write 3 New Done Done true 6032 Acked 7031 Write 9 New Done Done true 6033 Acked 7032 Write 5 New Done Done true 6034 Acked 7033 Write 2 New Done Done true 6035 Acked 7034 Read 2 New Done Done true 6036 Acked 7035 Read 3 New Done Done true 6037 Acked 7036 Read 1 New Done Done true 6038 Acked 7037 Flush 0 New Done Done true 6039 Acked 7038 Write 1 New Done Done true 6040 Acked 7039 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 6738 7039 7039 Downstairs last five completed: 6738 6737 6736 6735 6734 Upstairs last five completed: 6040 6039 6038 6037 6036 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:30.488009468Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:30.488076907Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:30.488104958Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:30.48813149Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:30.488151949Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:30.488186226Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:30.488446265Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:31.489198999Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:31.489263939Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:31.489291849Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:31.489315889Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:31.489341807Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:31.489386575Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:31.489531935Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:31.489961619Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1205) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.490784207Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.49093292Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.49289321Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.492923904Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.492947523Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 6738","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.49323259Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.493262535Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.493285845Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.493308225Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:32.49333046Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7039 7038 7037 7036 7035 Upstairs last five completed: 6040 6039 6038 6037 6036 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:34:34.054 INFO request completed, latency_us: 449, response_code: 204, uri:[8810] Got stop action so:false kr:false /stop/cid/0, method: POST, req_id: fd3003ab-0a71-41af-8a29-a4b4df85e128, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:34:34.055 INFO request completed, latency_us: 186, response_code: 200, uri: /state/cid/0, method: GET, req_id: 24e091ac-eae0-401e-b683-595c1e659a3a, remote_addr: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:34.055542227Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:34.055634115Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:34.055666672Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:34.05568776Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:34.055710511Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:34.055745593Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:35.056194128Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:35.056262762Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:35.056291174Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:35.05631779Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:35.056338681Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:35.056403851Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:35.056475929Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:36.057540536Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:36.057609574Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:36.057637638Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:36.057664359Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:36.057684619Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:36.057749785Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:36.057816708Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:37.058633964Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:37.058701988Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:37.058729902Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:37.058754416Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:37.058780788Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:37.058812449Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:37.058835932Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:37.058883174Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:34:38.058 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/0, method: GET, req_id: a8d9feb8-8e04-46c6-b21f-504ad1f290d2, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:38.05908033Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:38.059146733Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:38.059174935Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:38.059201511Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:38.059222137Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:38.059317443Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:38.059414551Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 001/300 Write block 1032 len 32768 data: 3 3 4 5 5 4 4 6 002/300 Write block 1738 len 4096 data: 4 003/300 Read block 2211 len 28672 004/300 Write block 4033 len 24576 data: 2 2 2 2 3 3 005/300 Write block 4331 len 32768 data: 2 3 5 5 6 6 7 6 006/300 Read block 4217 len 20480 007/300 Write block 3453 len 24576 data: 5 6 7 6 6 4 008/300 Write block 3828 len 28672 data: 4 4 4 3 3 3 3 009/300 Write block 288 len 12288 data: 5 5 5 010/300 Write block 624 len 20480 data: 3 3 3 3 4 011/300 Flush 012/300 Read block 1202 len 24576 013/300 Flush 014/300 Flush 015/300 Read block 1298 len 4096 016/300 Read block 1619 len 16384 017/300 Read block 3884 len 12288 018/300 Write block 4785 len 4096 data: 3 019/300 Read block 4071 len 16384 020/300 Read block 402 len 28672 021/300 Read block 4498 len 20480 022/300 Write block 3420 len 40960 data: 5 6 5 4 5 5 4 4 5 4 023/300 Write block 4681 len 36864 data: 2 3 3 3 3 4 3 3 3 024/300 Read block 2911 len 8192 025/300 Write block 225 len 20480 data: 5 4 5 4 3 026/300 Read block 2078 len 12288 027/300 Read block 26 len 32768 028/300 Write block 1626 len 40960 data: 7 7 4 6 6 7 5 5 5 7 029/300 Read block 121 len 12288 030/300 Flush 031/300 Read block 2885 len 28672 032/300 Write block 707 len 36864 data: 4 6 6 7 8 7 7 6 5 033/300 Read block 4026 len 4096 034/300 Read block 2804 len 12288 035/300 Flush 036/300 Write block 1295 len 16384 data: 5 7 7 7 037/300 Write block 3754 len 40960 data: 3 2 2 4 5 4 4 4 4 6 038/300 Read block 2384 len 8192 039/300 Write block 3788 len 28672 data: 6 6 5 6 6 4 4 040/300 Read block 2563 len 28672 041/300 Write block 4619 len 12288 data: 4 4 4 042/300 Read block 3582 len 8192 043/300 Read block 4264 len 8192 044/300 Read block 2261 len 24576 045/300 Flush 046/300 Read block 830 len 16384 047/300 Read block 4184 len 28672 048/300 Write block 659 len 28672 data: 10 11 10 10 12 12 11 049/300 Read block 4331 len 24576 050/300 Read block 3046 len 20480 051/300 Read block 677 len 28672 052/300 Read block 3434 len 12288 053/300 Read block 4262 len 36864 054/300 Write block 1248 len 24576 data: 4 4 3 2 2 5 055/300 Read block 647 len 40960 056/300 Write block 3995 len 36864 data: 9 10 8 6 3 3 3 3 6 057/300 Read block 958 len 12288 058/300 Read block 1809 len 20480 059/300 Write block 3877 len 28672 data: 3 3 3 3 6 7 7 060/300 Read block 2305 len 40960 061/300 Read block 4017 len 28672 062/300 Write block 3684 len 8192 data: 3 3 063/300 Write block 678 len 28672 data: 3 4 3 2 2 4 4 064/300 Write block 911 len 16384 data: 4 3 3 3 065/300 Read block 4035 len 16384 066/300 Read block 4155 len 8192 067/300 Write block 288 len 40960 data: 6 6 6 5 6 6 7 7 7 6 068/300 Write block 2174 len 40960 data: 3 3 3 2 2 2 3 2 2 2 069/300 Write block 3533 len 4096 data: 4 070/300 Read block 4884 len 40960 071/300 Flush 072/300 Read block 1304 len 12288 073/300 Read block 4195 len 24576 074/300 Flush 075/300 Read block 1106 len 40960 076/300 Read block 2975 len 28672 077/300 Write block 4320 len 40960 data: 6 6 6 6 6 6 6 6 4 4 078/300 Write block 3200 len 28672 data: 4 4 3 3 3 3 3 079/300 Read block 2719 len 28672 080/300 Read block 2106 len 24576 081/300 Write block 2511 len 36864 data: 10 7 7 7 6 7 7 7 5 082/300 Write block 3802 len 28672 data: 4 4 4 4 3 2 2 083/300 Write block 1457 len 20480 data: 4 4 4 5 6 084/300 Write block 4231 len 24576 data: 6 6 7 8 7 7 085/300 Write block 3665 len 32768 data: 4 5 5 5 4 4 3 3 086/300 Read block 152 len 12288 087/300 Write block 3812 len 24576 data: 2 3 3 3 3 4 088/300 Read block 584 len 28672 089/300 Flush 090/300 Flush 091/300 Write block 1617 len 32768 data: 3 2 2 2 3 3 3 5 092/300 Write block 2424 len 28672 data: 5 3 2 3 4 3 3 093/300 Write block 1243 len 16384 data: 6 6 5 5 094/300 Read block 4286 len 36864 095/300 Read block 2192 len 32768 096/300 Write block 2464 len 24576 data: 6 6 6 5 6 5 097/300 Flush 098/300 Read block 992 len 16384 099/300 Write block 1763 len 28672 data: 6 5 4 5 5 5 5 100/300 Read block 1049 len 32768 101/300 Write block 2480 len 16384 data: 6 4 4 6 102/300 Write block 2348 len 12288 data: 2 3 4 103/300 Read block 2298 len 28672 104/300 Read block 2328 len 24576 105/300 Read block 3551 len 32768 106/300 Write block 472 len 8192 data: 5 5 107/300 Write block 3063 len 36864 data: 7 7 4 5 5 2 2 2 3 108/300 Read block 979 len 32768 109/300 Read block 3306 len 24576 110/300 Read block 2944 len 12288 111/300 Read block 1434 len 20480 112/300 Write block 2626 len 20480 data: 6 8 9 8 7 113/300 Write block 929 len 40960 data: 5 4 3 5 4 5 4 5 6 7 114/300 Read block 1873 len 20480 115/300 Read block 2533 len 36864 116/300 Read block 1260 len 36864 117/300 Read block 2847 len 20480 118/300 Read block 2765 len 4096 119/300 Write block 1514 len 28672 data: 3 3 3 3 4 5 4 120/300 Write block 1000 len 4096 data: 4 121/300 Write block 4633 len 36864 data: 7 5 5 5 7 5 6 6 5 122/300 Write block 1832 len 32768 data: 5 3 3 2 2 2 2 2 123/300 Write block 4720 len 8192 data: 4 3 124/300 Write block 533 len 36864 data: 3 5 8 8 7 6 6 5 5 125/300 Write block 2799 len 8192 data: 6 6 126/300 Write block 3429 len 32768 data: 5 3 3 4 2 2 2 2 127/300 Read block 4167 len 28672 128/300 Read block 3766 len 28672 129/300 Write block 3468 len 20480 data: 3 3 5 4 4 130/300 Read block 3364 len 24576 131/300 Write block 3168 len 8192 data: 2 3 132/300 Write block 37 len 12288 data: 4 5 6 133/300 Read block 1487 len 20480 134/300 Flush 135/300 Write block 3453 len 36864 data: 6 7 8 7 7 5 4 3 3 136/300 Read block 1129 len 32768 137/300 Write block 3752 len 24576 data: 3 3 4 3 3 5 138/300 Flush 139/300 Write block 36 len 8192 data: 6 5 140/300 Read block 2117 len 32768 141/300 Read block 2282 len 4096 142/300 Write block 2225 len 40960 data: 2 3 4 4 4 4 4 3 4 5 143/300 Write block 1220 len 4096 data: 4 144/300 Write block 449 len 16384 data: 3 4 5 5 145/300 Read block 174 len 20480 146/300 Read block 4632 len 12288 147/300 Flush 148/300 Read block 1782 len 20480 149/300 Read block 3332 len 20480 150/300 Read block 2270 len 8192 151/300 Read block 4263 len 24576 152/300 Read block 3012 len 24576 153/300 Flush 154/300 Write block 976 len 8192 data: 6 5 155/300 Write block 4725 len 40960 data: 5 5 4 4 4 7 7 7 8 8 156/300 Read block 2451 len 24576 157/300 Read block 4554 len 32768 158/300 Write block 4150 len 16384 data: 4 4 3 3 159/300 Read block 1463 len 28672 160/300 Flush 161/300 Read block 415 len 32768 162/300 Write block 3979 len 20480 data: 5 5 4 3 4 163/300 Read block 2074 len 24576 164/300 Read block 3497 len 36864 165/300 Read block 1390 len 40960 166/300 Read block 3620 len 12288 167/300 Read block 4991 len 24576 168/300 Read block 1380 len 20480 169/300 Write block 3810 len 4096 data: 2 170/300 Read block 4171 len 4096 171/300 Read block 2729 len 40960 172/300 Read block 2513 len 8192 173/300 Read block 2534 len 8192 174/300 Write block 4248 len 8192 data: 8 8 175/300 Write block 1058 len 24576 data: 2 2 2 2 2 2 176/300 Write block 2176 len 32768 data: 4 3 3 3 4 3 3 3 177/300 Write block 3107 len 24576 data: 6 5 7 6 5 5 178/300 Read block 4614 len 28672 179/300 Write block 45 len 20480 data: 4 4 4 5 5 180/300 Read block 4134 len 16384 181/300 Read block 49 len 4096 182/300 Write block 783 len 36864 data: 4 3 2 2 2 2 2 2 2 183/300 Write block 1063 len 28672 data: 3 3 3 4 4 4 5 184/300 Write block 185 len 12288 data: 2 2 2 185/300 Write block 3114 len 36864 data: 4 6 4 4 4 4 5 5 5 186/300 Read block 1697 len 20480 187/300 Read block 1424 len 24576 188/300 Write block 403 len 12288 data: 5 5 3 189/300 Write block 3147 len 28672 data: 5 5 6 6 5 6 5 190/300 Read block 2498 len 32768 191/300 Read block 1787 len 16384 192/300 Read block 394 len 4096 193/300 Write block 3090 len 28672 data: 4 4 4 4 4 5 4 194/300 Write block 1640 len 20480 data: 5 5 4 4 5 195/300 Read block 4276 len 28672 196/300 Read block 4526 len 8192 197/300 Write block 424 len 40960 data: 4 3 4 5 6 7 6 7 7 6 198/300 Read block 3206 len 40960 199/300 Read block 4086 len 12288 200/300 Write block 1946 len 12288 data: 5 5 5 201/300 Read block 4316 len 12288 202/300 Write block 1143 len 28672 data: 5 3 4 3 3 2 3 203/300 Read block 1880 len 4096 204/300 Flush 205/300 Write block 2419 len 20480 data: 7 7 6 5 6 206/300 Write block 1745 len 20480 data: 5 5 5 4 4 207/300 Write block 15 len 8192 data: 8 8 208/300 Write block 547 len 20480 data: 5 5 5 5 4 209/300 Write block 4349 len 28672 data: 3 4 4 4 3 3 3 210/300 Write block 3283 len 16384 data: 2 2 2 2 211/300 Read block 4122 len 28672 212/300 Read block 3083 len 32768 213/300 Write block 1873 len 40960 data: 5 5 5 3 3 4 3 4 4 4 214/300 Write block 966 len 28672 data: 5 4 4 4 4 4 5 215/300 Write block 2329 len 8192 data: 9 7 216/300 Write block 1023 len 24576 data: 6 5 5 4 4 4 217/300 Read block 319 len 8192 218/300 Write block 3585 len 32768 data: 2 2 3 4 5 6 7 6 219/300 Read block 3537 len 36864 220/300 Write block 4318 len 4096 data: 5 221/300 Read block 45 len 8192 222/300 Flush 223/300 Read block 3615 len 28672 224/300 Flush 225/300 Write block 4268 len 20480 data: 11 11 9 6 6 226/300 Write block 4474 len 16384 data: 5 5 3 4 227/300 Read block 74 len 36864 228/300 Read block 1675 len 32768 229/300 Write block 4091 len 16384 data: 8 7 5 6 230/300 Write block 3581 len 8192 data: 2 2 231/300 Write block 3194 len 20480 data: 6 6 5 5 4 232/300 Write block 2320 len 24576 data: 4 4 3 3 4 4 233/300 Read block 3406 len 16384 234/300 Read block 338 len 12288 235/300 Flush 236/300 Read block 3422 len 8192 237/300 Read block 1544 len 36864 238/300 Write block 446 len 12288 data: 3 2 3 239/300 Flush 240/300 Write block 907 len 4096 data: 4 241/300 Write block 4737 len 24576 data: 9 9 7 8 7 6 242/300 Read block 596 len 24576 243/300 Write block 2950 len 12288 data: 9 9 7 244/300 Write block 319 len 4096 data: 8 245/300 Write block 1928 len 20480 data: 3 3 2 2 2 246/300 Write block 3290 len 28672 data: 2 2 2 2 2 2 2 247/300 Read block 4209 len 32768 248/300 Read block 4308 len 12288 249/300 Write block 1276 len 12288 data: 5 5 3 250/300 Read block 4567 len 20480 251/300 Write block 2402 len 28672 data: 4 3 3 3 4 5 5 252/300 Read block 2805 len 4096 253/300 Read block 2062 len 12288 254/300 Write block 989 len 36864 data: 3 3 3 3 3 3 3 3 3 255/300 Write block 3625 len 12288 data: 7 6 5 256/300 Flush 257/300 Write block 4751 len 16384 data: 4 2 3 3 258/300 Read block 1016 len 32768 259/300 Read block 1176 len 24576 260/300 Write block 2551 len 16384 data: 4 4 4 4 261/300 Read block 3610 len 28672 262/300 Read block 2232 len 24576 263/300 Flush 264/300 Read block 3151 len 12288 265/300 Read block 113 len 40960 266/300 Write block 624 len 24576 data: 4 4 4 4 5 4 267/300 Flush 268/300 Read block 158 len 8192 269/300 Flush 270/300 Read block 661 len 4096 271/300 Write block 4899 len 8192 data: 3 4 272/300 Read block 607 len 16384 273/300 Read block 4805 len 20480 274/300 Write block 531 len 20480 data: 4 5 4 6 9 275/300 Read block 3737 len 28672 276/300 Read block 3022 len 16384 277/300 Read block 3084 len 32768 278/300 Read block 1920 len 24576 279/300 Read block 1126 len 4096 280/300 Read block 728 len 4096 281/300 Read block 328 len 36864 282/300 Write block 1072 len 36864 data: 9 9 9 8 6 8 7 6 5 283/300 Read block 3384 len 36864 284/300 Flush 285/300 Write block 224 len 20480 data: 6 6 5 6 5 286/300 Write block 3102 len 4096 data: 5 287/300 Write block 97 len 40960 data: 6 6 6 6 6 4 5 5 7 7 288/300 Read block 2032 len 8192 289/300 Write block 137 len 20480 data: 5 5 4 3 3 290/300 Read block 651 len 4096 291/300 Read block 296 len 20480 292/300 Write block 1668 len 36864 data: 6 8 7 7 6 6 5 7 7 293/300 Flush 294/300 Read block 2015 len 8192 295/300 Read block 1473 len 36864 296/300 Read block 125 len 28672 297/300 Write block 4755 len 4096 data: 3 298/300 Write block 1152 len 12288 data: 2 3 3 299/300 Write block 3031 len 4096 data: 5 300/300 Write block 3034 len 4096 data: 4 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:34:38.640 INFO request completed, latency_us: 290, response_code: 204, uri: /start/cid/0, method: POST, req_id: 43432ecc-9465-4c3d-a5a3-b1452f38fded, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 6041 Acked 7040 Write 8 New Done Done false 6042 Acked 7041 Write 1 New Done Done false 6043 Acked 7042 Read 7 New Done Done false 6044 Acked 7043 Write 6 New Done Done false 6045 Acked 7044 Write 8 New Done Done false 6046 Acked 7045 Read 5 New Done Done false 6047 Acked 7046 Write 6 New Done Done false 6048 Acked 7047 Write 7 New Done Done false 6049 Acked 7048 Write 3 New Done Done false 6050 Acked 7049 Write 5 New Done Done false 6051 Acked 7050 Flush 0 New Done Done false 6052 Acked 7051 Read 6 New Done Done false 6053 Acked 7052 Flush 0 New Done Done false 6054 Acked 7053 Flush 0 New Done Done false 6055 Acked 7054 Read 1 New Done Done false 6056 Acked 7055 Read 4 New Done Done false 6057 Acked 7056 Read 3 New Done Done false 6058 Acked 7057 Write 1 New Done Done false 6059 Acked 7058 Read 4 New Done Done false 6060 Acked 7059 Read 7 New Done Done false 6061 Acked 7060 Read 5 New Done Done false 6062 Acked 7061 Write 10 New Done Done false 6063 Acked 7062 Write 9 New Done Done false 6064 Acked 7063 Read 2 New Done Done false 6065 Acked 7064 Write 5 New Done Done false 6066 Acked 7065 Read 3 New Done Done false 6067 Acked 7066 Read 8 New Done Done false 6068 Acked 7067 Write 10 New Done Done false 6069 Acked 7068 Read 3 New Done Done false 6070 Acked 7069 Flush 0 New Done Done false 6071 Acked 7070 Read 7 New Done Done false 6072 Acked 7071 Write 9 New Done Done false 6073 Acked 7072 Read 1 New Done Done false 6074 Acked 7073 Read 3 New Done Done false 6075 Acked 7074 Flush 0 New Done Done false 6076 Acked 7075 Write 4 New Done Done false 6077 Acked 7076 Write 10 New Done Done false 6078 Acked 7077 Read 2 New Done Done false 6079 Acked 7078 Write 7 New Done Done false 6080 Acked 7079 Read 7 New Done Done false 6081 Acked 7080 Write 3 New Done Done false 6082 Acked 7081 Read 2 New Done Done false 6083 Acked 7082 Read 2 New Done Done false 6084 Acked 7083 Read 6 New Done Done false 6085 Acked 7084 Flush 0 New Done Done false 6086 Acked 7085 Read 4 New Done Done false 6087 Acked 7086 Read 7 New Done Done false 6088 Acked 7087 Write 7 New Done Done false 6089 Acked 7088 Read 6 New Done Done false 6090 Acked 7089 Read 5 New Done Done false 6091 Acked 7090 Read 7 New Done Done false 6092 Acked 7091 Read 3 New Done Done false 6093 Acked 7092 Read 9 New Done Done false 6094 Acked 7093 Write 6 New Done Done false 6095 Acked 7094 Read 10 New Done Done false 6096 Acked 7095 Write 9 New Done Done false 6097 Acked 7096 Read 3 New Done Done false 6098 Acked 7097 Read 5 New Done Done false 6099 Acked 7098 Write 7 New Done Done false 6100 Acked 7099 Read 10 New Done Done false 6101 Acked 7100 Read 7 New Done Done false 6102 Acked 7101 Write 2 New Done Done false 6103 Acked 7102 Write 7 New Done Done false 6104 Acked 7103 Write 4 New Done Done false 6105 Acked 7104 Read 4 New Done Done false 6106 Acked 7105 Read 2 New Done Done false 6107 Acked 7106 Write 10 New Done Done false 6108 Acked 7107 Write 10 New Done Done false 6109 Acked 7108 Write 1 New Done Done false 6110 Acked 7109 Read 10 New Done Done false 6111 Acked 7110 Flush 0 New Done Done false 6112 Acked 7111 Read 3 New Done Done false 6113 Acked 7112 Read 6 New Done Done false 6114 Acked 7113 Flush 0 New Done Done false 6115 Acked 7114 Read 10 New Done Done false 6116 Acked 7115 Read 7 New Done Done false 6117 Acked 7116 Write 10 New Done Done false 6118 Acked 7117 Write 7 New Done Done false 6119 Acked 7118 Read 7 New Done Done false 6120 Acked 7119 Read 6 New Done Done false 6121 Acked 7120 Write 9 New Done Done false 6122 Acked 7121 Write 7 New Done Done false 6123 Acked 7122 Write 5 New Done Done false 6124 Acked 7123 Write 6 New Done Done false 6125 Acked 7124 Write 8 New Done Done false 6126 Acked 7125 Read 3 New Done Done false 6127 Acked 7126 Write 6 New Done Done false 6128 Acked 7127 Read 7 New Done Done false 6129 Acked 7128 Flush 0 New Done Done false 6130 Acked 7129 Flush 0 New Done Done false 6131 Acked 7130 Write 8 New Done Done false 6132 Acked 7131 Write 7 New Done Done false 6133 Acked 7132 Write 4 New Done Done false 6134 Acked 7133 Read 9 New Done Done false 6135 Acked 7134 Read 8 New Done Done false 6136 Acked 7135 Write 6 New Done Done false 6137 Acked 7136 Flush 0 New Done Done false 6138 Acked 7137 Read 4 New Done Done false 6139 Acked 7138 Write 7 New Done Done false 6140 Acked 7139 Read 8 New Done Done false 6141 Acked 7140 Write 4 New Done Done false 6142 Acked 7141 Write 3 New Done Done false 6143 Acked 7142 Read 7 New Done Done false 6144 Acked 7143 Read 6 New Done Done false 6145 Acked 7144 Read 8 New Done Done false 6146 Acked 7145 Write 2 New Done Done false 6147 Acked 7146 Write 9 New Done Done false 6148 Acked 7147 Read 8 New Done Done false 6149 Acked 7148 Read 6 New Done Done false 6150 Acked 7149 Read 3 New Done Done false 6151 Acked 7150 Read 5 New Done Done false 6152 Acked 7151 Write 5 New Done Done false 6153 Acked 7152 Write 10 New Done Done false 6154 Acked 7153 Read 5 New Done Done false 6155 Acked 7154 Read 9 New Done Done false 6156 Acked 7155 Read 9 New Done Done false 6157 Acked 7156 Read 5 New Done Done false 6158 Acked 7157 Read 1 New Done Done false 6159 Acked 7158 Write 7 New Done Done false 6160 Acked 7159 Write 1 New Done Done false 6161 Acked 7160 Write 9 New Done Done false 6162 Acked 7161 Write 8 New Done Done false 6163 Acked 7162 Write 2 New Done Done false 6164 Acked 7163 Write 9 New Done Done false 6165 Acked 7164 Write 2 New Done Done false 6166 Acked 7165 Write 8 New Done Done false 6167 Acked 7166 Read 7 New Done Done false 6168 Acked 7167 Read 7 New Done Done false 6169 Acked 7168 Flush 0 New Done Done false 6170 Acked 7169 Write 5 New Done Done false 6171 Acked 7170 Read 6 New Done Done false 6172 Acked 7171 Write 2 New Done Done false 6173 Acked 7172 Write 3 New Done Done false 6174 Acked 7173 Read 5 New Done Done false 6175 Acked 7174 Flush 0 New Done Done false 6176 Acked 7175 Write 9 New Done Done false 6177 Acked 7176 Read 8 New Done Done false 6178 Acked 7177 Write 6 New Done Done false 6179 Acked 7178 Flush 0 New Done Done false 6180 Acked 7179 Write 2 New Done Done false 6181 Acked 7180 Read 8 New Done Done false 6182 Acked 7181 Read 1 New Done Done false 6183 Acked 7182 Write 10 New Done Done false 6184 Acked 7183 Write 1 New Done Done false 6185 Acked 7184 Write 4 New Done Done false 6186 Acked 7185 Read 5 New Done Done false 6187 Acked 7186 Read 3 New Done Done false 6188 Acked 7187 Flush 0 New Done Done false 6189 Acked 7188 Read 5 New Done Done false 6190 Acked 7189 Read 5 New Done Done false 6191 Acked 7190 Read 2 New Done Done false 6192 Acked 7191 Read 6 New Done Done false 6193 Acked 7192 Read 6 New Done Done false 6194 Acked 7193 Flush 0 New Done Done false 6195 Acked 7194 Write 2 New Done Done false 6196 Acked 7195 Write 10 New Done Done false 6197 Acked 7196 Read 6 New Done Done false 6198 Acked 7197 Read 8 New Done Done false 6199 Acked 7198 Write 4 New Done Done false 6200 Acked 7199 Read 7 New Done Done false 6201 Acked 7200 Flush 0 New Done Done false 6202 Acked 7201 Read 8 New Done Done false 6203 Acked 7202 Write 5 New Done Done false 6204 Acked 7203 Read 6 New Done Done false 6205 Acked 7204 Read 9 New Done Done false 6206 Acked 7205 Read 10 New Done Done false 6207 Acked 7206 Read 3 New Done Done false 6208 Acked 7207 Read 6 New Done Done false 6209 Acked 7208 Read 5 New Done Done false 6210 Acked 7209 Write 1 New Done Done false 6211 Acked 7210 Read 1 New Done Done false 6212 Acked 7211 Read 10 New Done Done false 6213 Acked 7212 Read 2 New Done Done false 6214 Acked 7213 Read 2 New Done Done false 6215 Acked 7214 Write 2 New Done Done false 6216 Acked 7215 Write 6 New Done Done false 6217 Acked 7216 Write 8 New Done Done false 6218 Acked 7217 Write 6 New Done Done false 6219 Acked 7218 Read 7 New Done Done false 6220 Acked 7219 Write 5 New Done Done false 6221 Acked 7220 Read 4 New Done Done false 6222 Acked 7221 Read 1 New Done Done false 6223 Acked 7222 Write 9 New Done Done false 6224 Acked 7223 Write 7 New Done Done false 6225 Acked 7224 Write 3 New Done Done false 6226 Acked 7225 Write 9 New Done Done false 6227 Acked 7226 Read 5 New Done Done false 6228 Acked 7227 Read 6 New Done Done false 6229 Acked 7228 Write 3 New Done Done false 6230 Acked 7229 Write 7 New Done Done false 6231 Acked 7230 Read 8 New Done Done false 6232 Acked 7231 Read 4 New Done Done false 6233 Acked 7232 Read 1 New Done Done false 6234 Acked 7233 Write 7 New Done Done false 6235 Acked 7234 Write 5 New Done Done false 6236 Acked 7235 Read 7 New Done Done false 6237 Acked 7236 Read 2 New Done Done false 6238 Acked 7237 Write 10 New Done Done false 6239 Acked 7238 Read 10 New Done Done false 6240 Acked 7239 Read 3 New Done Done false 6241 Acked 7240 Write 3 New Done Done false 6242 Acked 7241 Read 3 New Done Done false 6243 Acked 7242 Write 7 New Done Done false 6244 Acked 7243 Read 1 New Done Done false 6245 Acked 7244 Flush 0 New Done Done false 6246 Acked 7245 Write 5 New Done Done false 6247 Acked 7246 Write 5 New Done Done false 6248 Acked 7247 Write 2 New Done Done false 6249 Acked 7248 Write 5 New Done Done false 6250 Acked 7249 Write 7 New Done Done false 6251 Acked 7250 Write 4 New Done Done false 6252 Acked 7251 Read 7 New Done Done false 6253 Acked 7252 Read 8 New Done Done false 6254 Acked 7253 Write 10 New Done Done false 6255 Acked 7254 Write 7 New Done Done false 6256 Acked 7255 Write 2 New Done Done false 6257 Acked 7256 Write 6 New Done Done false 6258 Acked 7257 Read 2 New Done Done false 6259 Acked 7258 Write 8 New Done Done false 6260 Acked 7259 Read 9 New Done Done false 6261 Acked 7260 Write 1 New Done Done false 6262 Acked 7261 Read 2 New Done Done false 6263 Acked 7262 Flush 0 New Done Done false 6264 Acked 7263 Read 7 New Done Done false 6265 Acked 7264 Flush 0 New Done Done false 6266 Acked 7265 Write 5 New Done Done false 6267 Acked 7266 Write 4 New Done Done false 6268 Acked 7267 Read 9 New Done Done false 6269 Acked 7268 Read 8 New Done Done false 6270 Acked 7269 Write 4 New Done Done false 6271 Acked 7270 Write 2 New Done Done false 6272 Acked 7271 Write 5 New Done Done false 6273 Acked 7272 Write 6 New Done Done false 6274 Acked 7273 Read 4 New Done Done false 6275 Acked 7274 Read 3 New Done Done false 6276 Acked 7275 Flush 0 New Done Done false 6277 Acked 7276 Read 2 New Done Done false 6278 Acked 7277 Read 9 New Done Done false 6279 Acked 7278 Write 3 New Done Done false 6280 Acked 7279 Flush 0 New Done Done false 6281 Acked 7280 Write 1 New Done Done false 6282 Acked 7281 Write 6 New Done Done false 6283 Acked 7282 Read 6 New Done Done false 6284 Acked 7283 Write 3 New Done Done false 6285 Acked 7284 Write 1 New Done Done false 6286 Acked 7285 Write 5 New Done Done false 6287 Acked 7286 Write 7 New Done Done false 6288 Acked 7287 Read 8 New Done Done false 6289 Acked 7288 Read 3 New Done Done false 6290 Acked 7289 Write 3 New Done Done false 6291 Acked 7290 Read 5 New Done Done false 6292 Acked 7291 Write 7 New Done Done false 6293 Acked 7292 Read 1 New Done Done false 6294 Acked 7293 Read 3 New Done Done false 6295 Acked 7294 Write 9 New Done Done false 6296 Acked 7295 Write 3 New Done Done false 6297 Acked 7296 Flush 0 New Done Done false 6298 Acked 7297 Write 4 New Done Done false 6299 Acked 7298 Read 8 New Done Done false 6300 Acked 7299 Read 6 New Done Done false 6301 Acked 7300 Write 4 New Done Done false 6302 Acked 7301 Read 7 New Done Done false 6303 Acked 7302 Read 6 New Done Done false 6304 Acked 7303 Flush 0 New Done Done false 6305 Acked 7304 Read 3 New Done Done false 6306 Acked 7305 Read 10 New Done Done false 6307 Acked 7306 Write 6 New Done Done false 6308 Acked 7307 Flush 0 New Done Done false 6309 Acked 7308 Read 2 New Done Done false 6310 Acked 7309 Flush 0 New Done Done false 6311 Acked 7310 Read 1 New Done Done false 6312 Acked 7311 Write 2 New Done Done false 6313 Acked 7312 Read 4 New Done Done false 6314 Acked 7313 Read 5 New Done Done false 6315 Acked 7314 Write 5 New Done Done false 6316 Acked 7315 Read 7 New Done Done false 6317 Acked 7316 Read 4 New Done Done false 6318 Acked 7317 Read 8 New Done Done false 6319 Acked 7318 Read 6 New Done Done false 6320 Acked 7319 Read 1 New Done Done false 6321 Acked 7320 Read 1 New Done Done false 6322 Acked 7321 Read 9 New Done Done false 6323 Acked 7322 Write 9 New Done Done false 6324 Acked 7323 Read 9 New Done Done false 6325 Acked 7324 Flush 0 New Done Done false 6326 Acked 7325 Write 5 New Done Done false 6327 Acked 7326 Write 1 New Done Done false 6328 Acked 7327 Write 10 New Done Done false 6329 Acked 7328 Read 2 New Done Done false 6330 Acked 7329 Write 5 New Done Done false 6331 Acked 7330 Read 1 New Done Done false 6332 Acked 7331 Read 5 New Done Done false 6333 Acked 7332 Write 9 New Done Done false 6334 Acked 7333 Flush 0 New Done Done false 6335 Acked 7334 Read 2 New Done Done false 6336 Acked 7335 Read 9 New Done Done false 6337 Acked 7336 Read 7 New Done Done false 6338 Acked 7337 Write 1 New Done Done false 6339 Acked 7338 Write 3 New Done Done false 6340 Acked 7339 Write 1 New Done Done false 6341 Acked 7340 Write 1 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7039 7333 7333 Downstairs last five completed: 7039 7038 7037 7036 7035 Upstairs last five completed: 6341 6340 6339 6338 6337 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:39.060745378Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:39.060885868Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:39.060920043Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:39.060947557Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:39.06096818Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:39.061025712Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:39.06128007Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:40.063112493Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:40.063176712Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:40.063204304Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:40.063230458Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:40.063250948Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:40.063358344Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:40.063557993Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:41.064554398Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:41.0646439Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:41.064672603Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:41.064699492Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:41.064719863Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:41.064763197Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:41.065042339Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:42.065703881Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:42.065792721Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:42.065822524Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:42.065846814Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:42.065872944Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:42.065893387Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:42.065926524Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:42.066160658Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 6041 Acked 7040 Write 8 New Done Done true 6042 Acked 7041 Write 1 New Done Done true 6043 Acked 7042 Read 7 New Done Done true 6044 Acked 7043 Write 6 New Done Done true 6045 Acked 7044 Write 8 New Done Done true 6046 Acked 7045 Read 5 New Done Done true 6047 Acked 7046 Write 6 New Done Done true 6048 Acked 7047 Write 7 New Done Done true 6049 Acked 7048 Write 3 New Done Done true 6050 Acked 7049 Write 5 New Done Done true 6051 Acked 7050 Flush 0 New Done Done true 6052 Acked 7051 Read 6 New Done Done true 6053 Acked 7052 Flush 0 New Done Done true 6054 Acked 7053 Flush 0 New Done Done true 6055 Acked 7054 Read 1 New Done Done true 6056 Acked 7055 Read 4 New Done Done true 6057 Acked 7056 Read 3 New Done Done true 6058 Acked 7057 Write 1 New Done Done true 6059 Acked 7058 Read 4 New Done Done true 6060 Acked 7059 Read 7 New Done Done true 6061 Acked 7060 Read 5 New Done Done true 6062 Acked 7061 Write 10 New Done Done true 6063 Acked 7062 Write 9 New Done Done true 6064 Acked 7063 Read 2 New Done Done true 6065 Acked 7064 Write 5 New Done Done true 6066 Acked 7065 Read 3 New Done Done true 6067 Acked 7066 Read 8 New Done Done true 6068 Acked 7067 Write 10 New Done Done true 6069 Acked 7068 Read 3 New Done Done true 6070 Acked 7069 Flush 0 New Done Done true 6071 Acked 7070 Read 7 New Done Done true 6072 Acked 7071 Write 9 New Done Done true 6073 Acked 7072 Read 1 New Done Done true 6074 Acked 7073 Read 3 New Done Done true 6075 Acked 7074 Flush 0 New Done Done true 6076 Acked 7075 Write 4 New Done Done true 6077 Acked 7076 Write 10 New Done Done true 6078 Acked 7077 Read 2 New Done Done true 6079 Acked 7078 Write 7 New Done Done true 6080 Acked 7079 Read 7 New Done Done true 6081 Acked 7080 Write 3 New Done Done true 6082 Acked 7081 Read 2 New Done Done true 6083 Acked 7082 Read 2 New Done Done true 6084 Acked 7083 Read 6 New Done Done true 6085 Acked 7084 Flush 0 New Done Done true 6086 Acked 7085 Read 4 New Done Done true 6087 Acked 7086 Read 7 New Done Done true 6088 Acked 7087 Write 7 New Done Done true 6089 Acked 7088 Read 6 New Done Done true 6090 Acked 7089 Read 5 New Done Done true 6091 Acked 7090 Read 7 New Done Done true 6092 Acked 7091 Read 3 New Done Done true 6093 Acked 7092 Read 9 New Done Done true 6094 Acked 7093 Write 6 New Done Done true 6095 Acked 7094 Read 10 New Done Done true 6096 Acked 7095 Write 9 New Done Done true 6097 Acked 7096 Read 3 New Done Done true 6098 Acked 7097 Read 5 New Done Done true 6099 Acked 7098 Write 7 New Done Done true 6100 Acked 7099 Read 10 New Done Done true 6101 Acked 7100 Read 7 New Done Done true 6102 Acked 7101 Write 2 New Done Done true 6103 Acked 7102 Write 7 New Done Done true 6104 Acked 7103 Write 4 New Done Done true 6105 Acked 7104 Read 4 New Done Done true 6106 Acked 7105 Read 2 New Done Done true 6107 Acked 7106 Write 10 New Done Done true 6108 Acked 7107 Write 10 New Done Done true 6109 Acked 7108 Write 1 New Done Done true 6110 Acked 7109 Read 10 New Done Done true 6111 Acked 7110 Flush 0 New Done Done true 6112 Acked 7111 Read 3 New Done Done true 6113 Acked 7112 Read 6 New Done Done true 6114 Acked 7113 Flush 0 New Done Done true 6115 Acked 7114 Read 10 New Done Done true 6116 Acked 7115 Read 7 New Done Done true 6117 Acked 7116 Write 10 New Done Done true 6118 Acked 7117 Write 7 New Done Done true 6119 Acked 7118 Read 7 New Done Done true 6120 Acked 7119 Read 6 New Done Done true 6121 Acked 7120 Write 9 New Done Done true 6122 Acked 7121 Write 7 New Done Done true 6123 Acked 7122 Write 5 New Done Done true 6124 Acked 7123 Write 6 New Done Done true 6125 Acked 7124 Write 8 New Done Done true 6126 Acked 7125 Read 3 New Done Done true 6127 Acked 7126 Write 6 New Done Done true 6128 Acked 7127 Read 7 New Done Done true 6129 Acked 7128 Flush 0 New Done Done true 6130 Acked 7129 Flush 0 New Done Done true 6131 Acked 7130 Write 8 New Done Done true 6132 Acked 7131 Write 7 New Done Done true 6133 Acked 7132 Write 4 New Done Done true 6134 Acked 7133 Read 9 New Done Done true 6135 Acked 7134 Read 8 New Done Done true 6136 Acked 7135 Write 6 New Done Done true 6137 Acked 7136 Flush 0 New Done Done true 6138 Acked 7137 Read 4 New Done Done true 6139 Acked 7138 Write 7 New Done Done true 6140 Acked 7139 Read 8 New Done Done true 6141 Acked 7140 Write 4 New Done Done true 6142 Acked 7141 Write 3 New Done Done true 6143 Acked 7142 Read 7 New Done Done true 6144 Acked 7143 Read 6 New Done Done true 6145 Acked 7144 Read 8 New Done Done true 6146 Acked 7145 Write 2 New Done Done true 6147 Acked 7146 Write 9 New Done Done true 6148 Acked 7147 Read 8 New Done Done true 6149 Acked 7148 Read 6 New Done Done true 6150 Acked 7149 Read 3 New Done Done true 6151 Acked 7150 Read 5 New Done Done true 6152 Acked 7151 Write 5 New Done Done true 6153 Acked 7152 Write 10 New Done Done true 6154 Acked 7153 Read 5 New Done Done true 6155 Acked 7154 Read 9 New Done Done true 6156 Acked 7155 Read 9 New Done Done true 6157 Acked 7156 Read 5 New Done Done true 6158 Acked 7157 Read 1 New Done Done true 6159 Acked 7158 Write 7 New Done Done true 6160 Acked 7159 Write 1 New Done Done true 6161 Acked 7160 Write 9 New Done Done true 6162 Acked 7161 Write 8 New Done Done true 6163 Acked 7162 Write 2 New Done Done true 6164 Acked 7163 Write 9 New Done Done true 6165 Acked 7164 Write 2 New Done Done true 6166 Acked 7165 Write 8 New Done Done true 6167 Acked 7166 Read 7 New Done Done true 6168 Acked 7167 Read 7 New Done Done true 6169 Acked 7168 Flush 0 New Done Done true 6170 Acked 7169 Write 5 New Done Done true 6171 Acked 7170 Read 6 New Done Done true 6172 Acked 7171 Write 2 New Done Done true 6173 Acked 7172 Write 3 New Done Done true 6174 Acked 7173 Read 5 New Done Done true 6175 Acked 7174 Flush 0 New Done Done true 6176 Acked 7175 Write 9 New Done Done true 6177 Acked 7176 Read 8 New Done Done true 6178 Acked 7177 Write 6 New Done Done true 6179 Acked 7178 Flush 0 New Done Done true 6180 Acked 7179 Write 2 New Done Done true 6181 Acked 7180 Read 8 New Done Done true 6182 Acked 7181 Read 1 New Done Done true 6183 Acked 7182 Write 10 New Done Done true 6184 Acked 7183 Write 1 New Done Done true 6185 Acked 7184 Write 4 New Done Done true 6186 Acked 7185 Read 5 New Done Done true 6187 Acked 7186 Read 3 New Done Done true 6188 Acked 7187 Flush 0 New Done Done true 6189 Acked 7188 Read 5 New Done Done true 6190 Acked 7189 Read 5 New Done Done true 6191 Acked 7190 Read 2 New Done Done true 6192 Acked 7191 Read 6 New Done Done true 6193 Acked 7192 Read 6 New Done Done true 6194 Acked 7193 Flush 0 New Done Done true 6195 Acked 7194 Write 2 New Done Done true 6196 Acked 7195 Write 10 New Done Done true 6197 Acked 7196 Read 6 New Done Done true 6198 Acked 7197 Read 8 New Done Done true 6199 Acked 7198 Write 4 New Done Done true 6200 Acked 7199 Read 7 New Done Done true 6201 Acked 7200 Flush 0 New Done Done true 6202 Acked 7201 Read 8 New Done Done true 6203 Acked 7202 Write 5 New Done Done true 6204 Acked 7203 Read 6 New Done Done true 6205 Acked 7204 Read 9 New Done Done true 6206 Acked 7205 Read 10 New Done Done true 6207 Acked 7206 Read 3 New Done Done true 6208 Acked 7207 Read 6 New Done Done true 6209 Acked 7208 Read 5 New Done Done true 6210 Acked 7209 Write 1 New Done Done true 6211 Acked 7210 Read 1 New Done Done true 6212 Acked 7211 Read 10 New Done Done true 6213 Acked 7212 Read 2 New Done Done true 6214 Acked 7213 Read 2 New Done Done true 6215 Acked 7214 Write 2 New Done Done true 6216 Acked 7215 Write 6 New Done Done true 6217 Acked 7216 Write 8 New Done Done true 6218 Acked 7217 Write 6 New Done Done true 6219 Acked 7218 Read 7 New Done Done true 6220 Acked 7219 Write 5 New Done Done true 6221 Acked 7220 Read 4 New Done Done true 6222 Acked 7221 Read 1 New Done Done true 6223 Acked 7222 Write 9 New Done Done true 6224 Acked 7223 Write 7 New Done Done true 6225 Acked 7224 Write 3 New Done Done true 6226 Acked 7225 Write 9 New Done Done true 6227 Acked 7226 Read 5 New Done Done true 6228 Acked 7227 Read 6 New Done Done true 6229 Acked 7228 Write 3 New Done Done true 6230 Acked 7229 Write 7 New Done Done true 6231 Acked 7230 Read 8 New Done Done true 6232 Acked 7231 Read 4 New Done Done true 6233 Acked 7232 Read 1 New Done Done true 6234 Acked 7233 Write 7 New Done Done true 6235 Acked 7234 Write 5 New Done Done true 6236 Acked 7235 Read 7 New Done Done true 6237 Acked 7236 Read 2 New Done Done true 6238 Acked 7237 Write 10 New Done Done true 6239 Acked 7238 Read 10 New Done Done true 6240 Acked 7239 Read 3 New Done Done true 6241 Acked 7240 Write 3 New Done Done true 6242 Acked 7241 Read 3 New Done Done true 6243 Acked 7242 Write 7 New Done Done true 6244 Acked 7243 Read 1 New Done Done true 6245 Acked 7244 Flush 0 New Done Done true 6246 Acked 7245 Write 5 New Done Done true 6247 Acked 7246 Write 5 New Done Done true 6248 Acked 7247 Write 2 New Done Done true 6249 Acked 7248 Write 5 New Done Done true 6250 Acked 7249 Write 7 New Done Done true 6251 Acked 7250 Write 4 New Done Done true 6252 Acked 7251 Read 7 New Done Done true 6253 Acked 7252 Read 8 New Done Done true 6254 Acked 7253 Write 10 New Done Done true 6255 Acked 7254 Write 7 New Done Done true 6256 Acked 7255 Write 2 New Done Done true 6257 Acked 7256 Write 6 New Done Done true 6258 Acked 7257 Read 2 New Done Done true 6259 Acked 7258 Write 8 New Done Done true 6260 Acked 7259 Read 9 New Done Done true 6261 Acked 7260 Write 1 New Done Done true 6262 Acked 7261 Read 2 New Done Done true 6263 Acked 7262 Flush 0 New Done Done true 6264 Acked 7263 Read 7 New Done Done true 6265 Acked 7264 Flush 0 New Done Done true 6266 Acked 7265 Write 5 New Done Done true 6267 Acked 7266 Write 4 New Done Done true 6268 Acked 7267 Read 9 New Done Done true 6269 Acked 7268 Read 8 New Done Done true 6270 Acked 7269 Write 4 New Done Done true 6271 Acked 7270 Write 2 New Done Done true 6272 Acked 7271 Write 5 New Done Done true 6273 Acked 7272 Write 6 New Done Done true 6274 Acked 7273 Read 4 New Done Done true 6275 Acked 7274 Read 3 New Done Done true 6276 Acked 7275 Flush 0 New Done Done true 6277 Acked 7276 Read 2 New Done Done true 6278 Acked 7277 Read 9 New Done Done true 6279 Acked 7278 Write 3 New Done Done true 6280 Acked 7279 Flush 0 New Done Done true 6281 Acked 7280 Write 1 New Done Done true 6282 Acked 7281 Write 6 New Done Done true 6283 Acked 7282 Read 6 New Done Done true 6284 Acked 7283 Write 3 New Done Done true 6285 Acked 7284 Write 1 New Done Done true 6286 Acked 7285 Write 5 New Done Done true 6287 Acked 7286 Write 7 New Done Done true 6288 Acked 7287 Read 8 New Done Done true 6289 Acked 7288 Read 3 New Done Done true 6290 Acked 7289 Write 3 New Done Done true 6291 Acked 7290 Read 5 New Done Done true 6292 Acked 7291 Write 7 New Done Done true 6293 Acked 7292 Read 1 New Done Done true 6294 Acked 7293 Read 3 New Done Done true 6295 Acked 7294 Write 9 New Done Done true 6296 Acked 7295 Write 3 New Done Done true 6297 Acked 7296 Flush 0 New Done Done true 6298 Acked 7297 Write 4 New Done Done true 6299 Acked 7298 Read 8 New Done Done true 6300 Acked 7299 Read 6 New Done Done true 6301 Acked 7300 Write 4 New Done Done true 6302 Acked 7301 Read 7 New Done Done true 6303 Acked 7302 Read 6 New Done Done true 6304 Acked 7303 Flush 0 New Done Done true 6305 Acked 7304 Read 3 New Done Done true 6306 Acked 7305 Read 10 New Done Done true 6307 Acked 7306 Write 6 New Done Done true 6308 Acked 7307 Flush 0 New Done Done true 6309 Acked 7308 Read 2 New Done Done true 6310 Acked 7309 Flush 0 New Done Done true 6311 Acked 7310 Read 1 New Done Done true 6312 Acked 7311 Write 2 New Done Done true 6313 Acked 7312 Read 4 New Done Done true 6314 Acked 7313 Read 5 New Done Done true 6315 Acked 7314 Write 5 New Done Done true 6316 Acked 7315 Read 7 New Done Done true 6317 Acked 7316 Read 4 New Done Done true 6318 Acked 7317 Read 8 New Done Done true 6319 Acked 7318 Read 6 New Done Done true 6320 Acked 7319 Read 1 New Done Done true 6321 Acked 7320 Read 1 New Done Done true 6322 Acked 7321 Read 9 New Done Done true 6323 Acked 7322 Write 9 New Done Done true 6324 Acked 7323 Read 9 New Done Done true 6325 Acked 7324 Flush 0 New Done Done true 6326 Acked 7325 Write 5 New Done Done true 6327 Acked 7326 Write 1 New Done Done true 6328 Acked 7327 Write 10 New Done Done true 6329 Acked 7328 Read 2 New Done Done true 6330 Acked 7329 Write 5 New Done Done true 6331 Acked 7330 Read 1 New Done Done true 6332 Acked 7331 Read 5 New Done Done true 6333 Acked 7332 Write 9 New Done Done true 6334 Acked 7333 Flush 0 New Done Done true 6335 Acked 7334 Read 2 New Done Done true 6336 Acked 7335 Read 9 New Done Done true 6337 Acked 7336 Read 7 New Done Done true 6338 Acked 7337 Write 1 New Done Done true 6339 Acked 7338 Write 3 New Done Done true 6340 Acked 7339 Write 1 New Done Done true 6341 Acked 7340 Write 1 New Done Done true 6342 Acked 7341 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7039 7341 7341 Downstairs last five completed: 7039 7038 7037 7036 7035 Upstairs last five completed: 6342 6341 6340 6339 6338 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:43.067278225Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:43.06734408Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:43.067379911Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:43.067407378Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:43.067428009Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:43.067450787Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:43.067694861Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:44.068538883Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:44.068605255Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:44.068633544Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:44.068660208Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:44.068680683Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:44.068841131Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:44.069046593Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1206) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.069906021Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.070082043Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.072063022Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.0720951Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.072119066Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 7039","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.072411522Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.072441065Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.072464329Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.072486944Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:45.072509198Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7341 7340 7339 7338 7337 Upstairs last five completed: 6342 6341 6340 6339 6338 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(1) stop 1 Jan 02 22:34:46.648[8820] Got stop action so:false kr:false INFO request completed, latency_us: 449, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e677382f-e3fd-4882-b6c5-a52e2b84c508, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:34:46.649 INFO request completed, latency_us: 395, response_code: 200, uri: /state/cid/1, method: GET, req_id: b4cdadf4-cd0d-4470-bcee-8be2a8dd13e0, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:46.650079827Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:46.650166765Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:46.650217967Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:46.650251985Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:46.650287952Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:46.650322581Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:47.651244124Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:47.651312132Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:47.651340707Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:47.651367145Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:47.651387594Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:47.651411637Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:47.651515452Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:48.652672465Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:48.652742621Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:48.652770879Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:48.652807203Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:48.652828641Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:48.652871296Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:48.652980833Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:49.654925582Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:49.65502978Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:49.65505786Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:49.655084559Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:49.655105015Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 0 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:49.65512942Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:49.655192912Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:34:50.652 INFO request completed, latency_us: 259, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4c39f5a7-bd2a-4591-bbe3-9805e2b3e6d7, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 2911 len 36864 002/300 Read block 2305 len 4096 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:50.656727216Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:50.656801279Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:50.656830523Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:50.656857145Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:50.656877631Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 2 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:50.656923887Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:50.657061388Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Flush 004/300 Write block 4771 len 36864 data: 6 6 5 4 4 3 2 3 3 005/300 Write block 866 len 12288 data: 7 7 5 006/300 Write block 4240 len 32768 data: 4 4 3 3 4 5 5 5 007/300 Write block 3565 len 20480 data: 2 2 2 2 3 008/300 Write block 634 len 8192 data: 5 5 009/300 Write block 3192 len 20480 data: 5 6 7 7 6 010/300 Read block 4101 len 32768 011/300 Write block 3960 len 20480 data: 7 6 6 5 5 012/300 Read block 3590 len 36864 013/300 Flush 014/300 Read block 4090 len 12288 015/300 Write block 4129 len 4096 data: 4 016/300 Read block 2452 len 20480 017/300 Write block 4963 len 12288 data: 7 7 7 018/300 Read block 3315 len 40960 019/300 Read block 1606 len 40960 020/300 Read block 1525 len 20480 021/300 Write block 1727 len 20480 data: 3 3 3 4 4 022/300 Read block 4668 len 40960 023/300 Read block 3226 len 4096 024/300 Read block 3382 len 24576 025/300 Read block 1759 len 4096 026/300 Flush 027/300 Flush 028/300 Write block 4772 len 36864 data: 7 6 5 5 4 3 4 4 3 029/300 Write block 3580 len 4096 data: 2 030/300 Flush 031/300 Read block 2379 len 36864 032/300 Read block 3086 len 28672 033/300 Write block 3129 len 20480 data: 4 3 2 4 5 034/300 Write block 3239 len 16384 data: 7 7 7 6 035/300 Write block 3761 len 28672 data: 5 5 7 7 9 10 8 036/300 Read block 2354 len 12288 037/300 Write block 2943 len 36864 data: 3 4 5 5 6 8 9 10 10 038/300 Read block 3001 len 20480 039/300 Write block 3098 len 28672 data: 5 6 5 5 6 5 5 040/300 Read block 4256 len 32768 041/300 Read block 4046 len 12288 042/300 Write block 4100 len 12288 data: 5 5 5 043/300 Write block 4385 len 8192 data: 4 4 044/300 Read block 2410 len 40960 045/300 Read block 3606 len 4096 046/300 Read block 1210 len 8192 047/300 Write block 729 len 16384 data: 7 6 7 6 048/300 Write block 3054 len 32768 data: 5 6 7 7 6 7 7 5 049/300 Flush 050/300 Write block 3304 len 24576 data: 6 6 6 5 4 5 051/300 Read block 356 len 28672 052/300 Read block 4893 len 28672 053/300 Read block 3730 len 36864 054/300 Read block 4061 len 16384 055/300 Write block 2953 len 4096 data: 8 056/300 Read block 2819 len 8192 057/300 Read block 932 len 40960 058/300 Write block 4916 len 4096 data: 3 059/300 Write block 4435 len 16384 data: 5 5 6 6 060/300 Read block 2494 len 8192 061/300 Write block 4856 len 40960 data: 2 2 2 3 3 4 2 2 2 2 062/300 Read block 1817 len 32768 063/300 Read block 4581 len 8192 064/300 Read block 4090 len 24576 065/300 Read block 2326 len 24576 066/300 Flush 067/300 Flush 068/300 Write block 4326 len 24576 data: 7 7 5 5 4 3 069/300 Read block 624 len 8192 070/300 Write block 2834 len 20480 data: 4 6 6 6 8 071/300 Read block 3861 len 20480 072/300 Read block 1154 len 12288 073/300 Write block 3833 len 32768 data: 4 4 3 3 4 4 5 5 074/300 Read block 2348 len 20480 075/300 Write block 2385 len 20480 data: 5 5 6 6 7 076/300 Read block 1488 len 24576 077/300 Write block 674 len 28672 data: 3 4 4 3 4 5 4 078/300 Write block 1911 len 12288 data: 4 5 4 079/300 Write block 4020 len 4096 data: 3 080/300 Read block 4296 len 24576 081/300 Write block 2680 len 12288 data: 3 3 4 082/300 Read block 739 len 16384 083/300 Write block 1656 len 20480 data: 6 6 5 5 7 084/300 Write block 3953 len 12288 data: 3 4 4 085/300 Write block 521 len 24576 data: 5 5 4 5 7 7 086/300 Flush 087/300 Flush 088/300 Write block 556 len 8192 data: 3 3 089/300 Read block 3696 len 24576 090/300 Flush 091/300 Write block 4207 len 4096 data: 5 092/300 Write block 2121 len 24576 data: 5 6 6 4 3 4 093/300 Read block 491 len 16384 094/300 Read block 1473 len 36864 095/300 Read block 4149 len 36864 096/300 Read block 1139 len 28672 097/300 Read block 2417 len 24576 098/300 Flush 099/300 Flush 100/300 Read block 497 len 32768 101/300 Flush 102/300 Read block 4938 len 24576 103/300 Read block 3680 len 8192 104/300 Read block 143 len 28672 105/300 Read block 4254 len 20480 106/300 Write block 2877 len 16384 data: 3 3 3 4 107/300 Read block 2014 len 40960 108/300 Read block 3168 len 24576 109/300 Write block 1753 len 24576 data: 4 5 7 7 8 8 110/300 Write block 2104 len 8192 data: 4 4 111/300 Read block 1094 len 16384 112/300 Read block 1089 len 36864 113/300 Read block 4474 len 40960 114/300 Write block 4752 len 8192 data: 3 4 115/300 Write block 655 len 36864 data: 5 8 8 10 11 12 11 11 13 116/300 Read block 1088 len 8192 117/300 Write block 280 len 20480 data: 5 5 5 7 6 118/300 Write block 1253 len 16384 data: 6 5 6 7 119/300 Flush 120/300 Write block 3699 len 16384 data: 5 5 5 5 121/300 Read block 3507 len 40960 122/300 Flush 123/300 Write block 3663 len 24576 data: 5 4 5 6 6 6 124/300 Read block 2507 len 16384 125/300 Read block 1479 len 40960 126/300 Read block 1723 len 28672 127/300 Read block 1476 len 32768 128/300 Read block 2281 len 20480 129/300 Write block 2529 len 40960 data: 4 4 4 5 5 4 4 4 3 3 130/300 Read block 3709 len 20480 131/300 Read block 1364 len 4096 132/300 Write block 3853 len 36864 data: 2 2 3 3 5 4 4 4 5 133/300 Read block 3255 len 12288 134/300 Read block 980 len 36864 135/300 Read block 920 len 36864 136/300 Read block 4395 len 32768 137/300 Flush 138/300 Read block 1860 len 8192 139/300 Read block 3831 len 36864 140/300 Write block 3401 len 20480 data: 4 4 3 3 3 141/300 Write block 788 len 16384 data: 3 3 3 3 142/300 Read block 776 len 24576 143/300 Read block 2432 len 8192 144/300 Read block 3425 len 4096 145/300 Write block 4595 len 24576 data: 4 4 4 3 3 4 146/300 Read block 1158 len 36864 147/300 Write block 72 len 4096 data: 2 148/300 Read block 462 len 32768 149/300 Write block 4224 len 8192 data: 5 5 150/300 Write block 2071 len 36864 data: 2 3 3 4 4 4 6 6 5 151/300 Read block 1299 len 36864 152/300 Write block 2755 len 36864 data: 6 4 3 3 3 4 4 5 5 153/300 Read block 1341 len 16384 154/300 Read block 4977 len 16384 155/300 Read block 366 len 16384 156/300 Read block 3263 len 16384 157/300 Write block 2948 len 28672 data: 9 10 11 11 8 9 7 158/300 Write block 3220 len 16384 data: 4 3 3 3 159/300 Read block 1243 len 8192 160/300 Write block 692 len 32768 data: 4 4 4 4 5 3 4 4 161/300 Read block 1931 len 28672 162/300 Read block 2948 len 24576 163/300 Write block 2446 len 32768 data: 7 7 5 4 4 4 2 2 164/300 Read block 1917 len 36864 165/300 Read block 916 len 24576 166/300 Read block 1921 len 36864 167/300 Flush 168/300 Write block 683 len 12288 data: 5 5 5 169/300 Write block 2195 len 28672 data: 5 7 6 6 5 5 5 170/300 Read block 1379 len 24576 171/300 Flush 172/300 Read block 1179 len 20480 173/300 Write block 4009 len 4096 data: 10 174/300 Write block 2883 len 16384 data: 3 2 2 4 175/300 Read block 2337 len 24576 176/300 Write block 3803 len 8192 data: 5 5 177/300 Write block 4802 len 16384 data: 6 8 8 9 178/300 Flush 179/300 Write block 2644 len 12288 data: 9 7 6 180/300 Write block 3489 len 16384 data: 3 2 2 2 181/300 Flush 182/300 Read block 1356 len 32768 183/300 Read block 4198 len 40960 184/300 Read block 1163 len 16384 185/300 Flush 186/300 Write block 2597 len 12288 data: 4 4 5 187/300 Read block 1458 len 40960 188/300 Read block 4316 len 28672 189/300 Write block 483 len 40960 data: 3 5 5 4 4 4 4 4 4 5 190/300 Write block 832 len 40960 data: 4 4 6 7 3 3 2 2 2 2 191/300 Write block 140 len 16384 data: 4 4 3 3 192/300 Read block 2169 len 40960 193/300 Write block 315 len 24576 data: 7 7 6 8 9 6 194/300 Write block 3455 len 8192 data: 9 8 195/300 Read block 3107 len 36864 196/300 Read block 2484 len 32768 197/300 Read block 3809 len 36864 198/300 Write block 1161 len 20480 data: 5 7 7 6 6 199/300 Write block 127 len 4096 data: 4 200/300 Write block 3789 len 28672 data: 7 6 7 7 5 5 4 201/300 Read block 2218 len 4096 202/300 Write block 3660 len 16384 data: 5 5 4 6 203/300 Read block 0 len 24576 204/300 Write block 544 len 4096 data: 3 205/300 Write block 2188 len 12288 data: 3 4 4 206/300 Read block 2284 len 8192 207/300 Write block 1317 len 24576 data: 3 4 4 3 4 6 208/300 Write block 4385 len 12288 data: 5 5 3 209/300 Read block 417 len 4096 210/300 Write block 3334 len 32768 data: 6 6 6 5 4 5 6 6 211/300 Write block 1166 len 36864 data: 7 5 5 6 5 3 2 2 2 212/300 Read block 1012 len 36864 213/300 Write block 774 len 8192 data: 4 3 214/300 Flush 215/300 Read block 4537 len 16384 216/300 Write block 3427 len 12288 data: 5 6 6 217/300 Read block 570 len 16384 218/300 Read block 4210 len 36864 219/300 Write block 1874 len 24576 data: 6 6 4 4 5 4 220/300 Read block 215 len 36864 221/300 Write block 4997 len 8192 data: 2 2 222/300 Read block 1115 len 8192 223/300 Read block 3215 len 12288 224/300 Write block 2777 len 32768 data: 5 4 4 5 4 4 4 4 225/300 Read block 3210 len 40960 226/300 Read block 4544 len 32768 227/300 Write block 1991 len 24576 data: 3 2 2 2 3 3 228/300 Read block 2932 len 8192 229/300 Read block 1667 len 16384 230/300 Flush 231/300 Write block 3089 len 8192 data: 4 5 232/300 Read block 188 len 12288 233/300 Flush 234/300 Write block 1100 len 16384 data: 5 5 6 7 235/300 Read block 1412 len 20480 236/300 Read block 2923 len 24576 237/300 Read block 3375 len 28672 238/300 Write block 3602 len 28672 data: 6 6 5 5 6 5 6 239/300 Write block 2132 len 12288 data: 3 3 3 240/300 Read block 4408 len 40960 241/300 Write block 4394 len 16384 data: 3 3 2 3 242/300 Read block 4582 len 24576 243/300 Read block 4951 len 16384 244/300 Flush 245/300 Read block 2033 len 12288 246/300 Write block 1799 len 36864 data: 3 3 3 3 3 3 4 4 4 247/300 Read block 92 len 28672 248/300 Write block 4269 len 16384 data: 12 10 7 7 249/300 Read block 3831 len 12288 250/300 Read block 2587 len 28672 251/300 Write block 2872 len 32768 data: 4 4 4 3 3 4 4 4 252/300 Write block 2390 len 36864 data: 7 8 6 6 5 4 4 3 3 253/300 Read block 280 len 32768 254/300 Write block 4597 len 28672 data: 5 4 4 5 4 4 6 255/300 Flush 256/300 Write block 2366 len 24576 data: 5 3 2 2 2 3 257/300 Read block 4725 len 40960 258/300 Read block 1276 len 8192 259/300 Flush 260/300 Flush 261/300 Write block 4192 len 36864 data: 3 3 4 3 2 2 2 2 3 262/300 Read block 3168 len 16384 263/300 Write block 4456 len 32768 data: 4 4 4 4 3 3 4 2 264/300 Read block 1356 len 24576 265/300 Write block 2072 len 24576 data: 4 4 5 5 5 7 266/300 Write block 4869 len 12288 data: 6 5 5 267/300 Read block 4165 len 40960 268/300 Read block 3492 len 24576 269/300 Read block 2955 len 36864 270/300 Write block 349 len 36864 data: 3 3 3 3 4 4 3 2 2 271/300 Read block 1190 len 40960 272/300 Read block 2255 len 32768 273/300 Flush 274/300 Write block 2258 len 4096 data: 6 275/300 Write block 4921 len 24576 data: 5 4 6 8 9 10 276/300 Read block 1506 len 40960 277/300 Write block 3247 len 20480 data: 7 7 7 6 6 278/300 Flush 279/300 Read block 1119 len 36864 280/300 Read block 3270 len 28672 281/300 Read block 1363 len 36864 282/300 Flush 283/300 Write block 1287 len 4096 data: 6 284/300 Read block 2553 len 40960 285/300 Read block 1128 len 36864 286/300 Flush 287/300 Read block 1905 len 24576 288/300 Write block 4413 len 12288 data: 2 2 2 289/300 Write block 3060 len 40960 data: 8 6 7 8 8 5 6 6 3 3 290/300 Read block 4042 len 32768 291/300 Read block 3750 len 12288 292/300 Write block 4148 len 32768 data: 7 5 5 5 4 4 5 5 293/300 Write block 2272 len 16384 data: 5 5 5 5 294/300 Flush 295/300 Write block 3269 len 32768 data: 4 2 2 2 2 2 3 3 296/300 Read block 3446 len 40960 297/300 Read block 1506 len 4096 298/300 Write block 3540 len 32768 data: 7 8 8 8 7 5 4 4 299/300 Flush 300/300 Read block 1098 len 40960 Main task has work to do, go find it got dsc Start(1) start 1 Jan 02 22:34:51.280 INFO request completed, latency_us: 293, response_code: 204, uri: /start/cid/1, method: POST, req_id: 84e81059-0494-48c5-8f45-80effbb7387f, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 1, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 6343 Acked 7342 Read 9 Done New Done true 6344 Acked 7343 Read 1 Done New Done true 6345 Acked 7344 Flush 0 Done New Done false 6346 Acked 7345 Write 9 Done New Done false 6347 Acked 7346 Write 3 Done New Done false 6348 Acked 7347 Write 8 Done New Done false 6349 Acked 7348 Write 5 Done New Done false 6350 Acked 7349 Write 2 Done New Done false 6351 Acked 7350 Write 5 Done New Done false 6352 Acked 7351 Read 8 Done New Done false 6353 Acked 7352 Write 5 Done New Done false 6354 Acked 7353 Read 9 Done New Done false 6355 Acked 7354 Flush 0 Done New Done false 6356 Acked 7355 Read 3 Done New Done false 6357 Acked 7356 Write 1 Done New Done false 6358 Acked 7357 Read 5 Done New Done false 6359 Acked 7358 Write 3 Done New Done false 6360 Acked 7359 Read 10 Done New Done false 6361 Acked 7360 Read 10 Done New Done false 6362 Acked 7361 Read 5 Done New Done false 6363 Acked 7362 Write 5 Done New Done false 6364 Acked 7363 Read 10 Done New Done false 6365 Acked 7364 Read 1 Done New Done false 6366 Acked 7365 Read 6 Done New Done false 6367 Acked 7366 Read 1 Done New Done false 6368 Acked 7367 Flush 0 Done New Done false 6369 Acked 7368 Flush 0 Done New Done false 6370 Acked 7369 Write 9 Done New Done false 6371 Acked 7370 Write 1 Done New Done false 6372 Acked 7371 Flush 0 Done New Done false 6373 Acked 7372 Read 9 Done New Done false 6374 Acked 7373 Read 7 Done New Done false 6375 Acked 7374 Write 5 Done New Done false 6376 Acked 7375 Write 4 Done New Done false 6377 Acked 7376 Write 7 Done New Done false 6378 Acked 7377 Read 3 Done New Done false 6379 Acked 7378 Write 9 Done New Done false 6380 Acked 7379 Read 5 Done New Done false 6381 Acked 7380 Write 7 Done New Done false 6382 Acked 7381 Read 8 Done New Done false 6383 Acked 7382 Read 3 Done New Done false 6384 Acked 7383 Write 3 Done New Done false 6385 Acked 7384 Write 2 Done New Done false 6386 Acked 7385 Read 10 Done New Done false 6387 Acked 7386 Read 1 Done New Done false 6388 Acked 7387 Read 2 Done New Done false 6389 Acked 7388 Write 4 Done New Done false 6390 Acked 7389 Write 8 Done New Done false 6391 Acked 7390 Flush 0 Done New Done false 6392 Acked 7391 Write 6 Done New Done false 6393 Acked 7392 Read 7 Done New Done false 6394 Acked 7393 Read 7 Done New Done false 6395 Acked 7394 Read 9 Done New Done false 6396 Acked 7395 Read 4 Done New Done false 6397 Acked 7396 Write 1 Done New Done false 6398 Acked 7397 Read 2 Done New Done false 6399 Acked 7398 Read 10 Done New Done false 6400 Acked 7399 Write 1 Done New Done false 6401 Acked 7400 Write 4 Done New Done false 6402 Acked 7401 Read 2 Done New Done false 6403 Acked 7402 Write 10 Done New Done false 6404 Acked 7403 Read 8 Done New Done false 6405 Acked 7404 Read 2 Done New Done false 6406 Acked 7405 Read 6 Done New Done false 6407 Acked 7406 Read 6 Done New Done false 6408 Acked 7407 Flush 0 Done New Done false 6409 Acked 7408 Flush 0 Done New Done false 6410 Acked 7409 Write 6 Done New Done false 6411 Acked 7410 Read 2 Done New Done false 6412 Acked 7411 Write 5 Done New Done false 6413 Acked 7412 Read 5 Done New Done false 6414 Acked 7413 Read 3 Done New Done false 6415 Acked 7414 Write 8 Done New Done false 6416 Acked 7415 Read 5 Done New Done false 6417 Acked 7416 Write 5 Done New Done false 6418 Acked 7417 Read 6 Done New Done false 6419 Acked 7418 Write 7 Done New Done false 6420 Acked 7419 Write 3 Done New Done false 6421 Acked 7420 Write 1 Done New Done false 6422 Acked 7421 Read 6 Done New Done false 6423 Acked 7422 Write 3 Done New Done false 6424 Acked 7423 Read 4 Done New Done false 6425 Acked 7424 Write 5 Done New Done false 6426 Acked 7425 Write 3 Done New Done false 6427 Acked 7426 Write 6 Done New Done false 6428 Acked 7427 Flush 0 Done New Done false 6429 Acked 7428 Flush 0 Done New Done false 6430 Acked 7429 Write 2 Done New Done false 6431 Acked 7430 Read 6 Done New Done false 6432 Acked 7431 Flush 0 Done New Done false 6433 Acked 7432 Write 1 Done New Done false 6434 Acked 7433 Write 6 Done New Done false 6435 Acked 7434 Read 4 Done New Done false 6436 Acked 7435 Read 9 Done New Done false 6437 Acked 7436 Read 9 Done New Done false 6438 Acked 7437 Read 7 Done New Done false 6439 Acked 7438 Read 6 Done New Done false 6440 Acked 7439 Flush 0 Done New Done false 6441 Acked 7440 Flush 0 Done New Done false 6442 Acked 7441 Read 8 Done New Done false 6443 Acked 7442 Flush 0 Done New Done false 6444 Acked 7443 Read 6 Done New Done false 6445 Acked 7444 Read 2 Done New Done false 6446 Acked 7445 Read 7 Done New Done false 6447 Acked 7446 Read 5 Done New Done false 6448 Acked 7447 Write 4 Done New Done false 6449 Acked 7448 Read 10 Done New Done false 6450 Acked 7449 Read 6 Done New Done false 6451 Acked 7450 Write 6 Done New Done false 6452 Acked 7451 Write 2 Done New Done false 6453 Acked 7452 Read 4 Done New Done false 6454 Acked 7453 Read 9 Done New Done false 6455 Acked 7454 Read 10 Done New Done false 6456 Acked 7455 Write 2 Done New Done false 6457 Acked 7456 Write 9 Done New Done false 6458 Acked 7457 Read 2 Done New Done false 6459 Acked 7458 Write 5 Done New Done false 6460 Acked 7459 Write 4 Done New Done false 6461 Acked 7460 Flush 0 Done New Done false 6462 Acked 7461 Write 4 Done New Done false 6463 Acked 7462 Read 10 Done New Done false 6464 Acked 7463 Flush 0 Done New Done false 6465 Acked 7464 Write 6 Done New Done false 6466 Acked 7465 Read 4 Done New Done false 6467 Acked 7466 Read 10 Done New Done false 6468 Acked 7467 Read 7 Done New Done false 6469 Acked 7468 Read 8 Done New Done false 6470 Acked 7469 Read 5 Done New Done false 6471 Acked 7470 Write 10 Done New Done false 6472 Acked 7471 Read 5 Done New Done false 6473 Acked 7472 Read 1 Done New Done false 6474 Acked 7473 Write 9 Done New Done false 6475 Acked 7474 Read 3 Done New Done false 6476 Acked 7475 Read 9 Done New Done false 6477 Acked 7476 Read 9 Done New Done false 6478 Acked 7477 Read 8 Done New Done false 6479 Acked 7478 Flush 0 Done New Done false 6480 Acked 7479 Read 2 Done New Done false 6481 Acked 7480 Read 9 Done New Done false 6482 Acked 7481 Write 5 Done New Done false 6483 Acked 7482 Write 4 Done New Done false 6484 Acked 7483 Read 6 Done New Done false 6485 Acked 7484 Read 2 Done New Done false 6486 Acked 7485 Read 1 Done New Done false 6487 Acked 7486 Write 6 Done New Done false 6488 Acked 7487 Read 9 Done New Done false 6489 Acked 7488 Write 1 Done New Done false 6490 Acked 7489 Read 8 Done New Done false 6491 Acked 7490 Write 2 Done New Done false 6492 Acked 7491 Write 9 Done New Done false 6493 Acked 7492 Read 9 Done New Done false 6494 Acked 7493 Write 9 Done New Done false 6495 Acked 7494 Read 4 Done New Done false 6496 Acked 7495 Read 4 Done New Done false 6497 Acked 7496 Read 4 Done New Done false 6498 Acked 7497 Read 4 Done New Done false 6499 Acked 7498 Write 7 Done New Done false 6500 Acked 7499 Write 4 Done New Done false 6501 Acked 7500 Read 2 Done New Done false 6502 Acked 7501 Write 8 Done New Done false 6503 Acked 7502 Read 7 Done New Done false 6504 Acked 7503 Read 6 Done New Done false 6505 Acked 7504 Write 8 Done New Done false 6506 Acked 7505 Read 9 Done New Done false 6507 Acked 7506 Read 6 Done New Done false 6508 Acked 7507 Read 9 Done New Done false 6509 Acked 7508 Flush 0 Done New Done false 6510 Acked 7509 Write 3 Done New Done false 6511 Acked 7510 Write 7 Done New Done false 6512 Acked 7511 Read 6 Done New Done false 6513 Acked 7512 Flush 0 Done New Done false 6514 Acked 7513 Read 5 Done New Done false 6515 Acked 7514 Write 1 Done New Done false 6516 Acked 7515 Write 4 Done New Done false 6517 Acked 7516 Read 6 Done New Done false 6518 Acked 7517 Write 2 Done New Done false 6519 Acked 7518 Write 4 Done New Done false 6520 Acked 7519 Flush 0 Done New Done false 6521 Acked 7520 Write 3 Done New Done false 6522 Acked 7521 Write 4 Done New Done false 6523 Acked 7522 Flush 0 Done New Done false 6524 Acked 7523 Read 8 Done New Done false 6525 Acked 7524 Read 10 Done New Done false 6526 Acked 7525 Read 4 Done New Done false 6527 Acked 7526 Flush 0 Done New Done false 6528 Acked 7527 Write 3 Done New Done false 6529 Acked 7528 Read 10 Done New Done false 6530 Acked 7529 Read 7 Done New Done false 6531 Acked 7530 Write 10 Done New Done false 6532 Acked 7531 Write 10 Done New Done false 6533 Acked 7532 Write 4 Done New Done false 6534 Acked 7533 Read 10 Done New Done false 6535 Acked 7534 Write 6 Done New Done false 6536 Acked 7535 Write 2 Done New Done false 6537 Acked 7536 Read 9 Done New Done false 6538 Acked 7537 Read 8 Done New Done false 6539 Acked 7538 Read 9 Done New Done false 6540 Acked 7539 Write 5 Done New Done false 6541 Acked 7540 Write 1 Done New Done false 6542 Acked 7541 Write 7 Done New Done false 6543 Acked 7542 Read 1 Done New Done false 6544 Acked 7543 Write 4 Done New Done false 6545 Acked 7544 Read 6 Done New Done false 6546 Acked 7545 Write 1 Done New Done false 6547 Acked 7546 Write 3 Done New Done false 6548 Acked 7547 Read 2 Done New Done false 6549 Acked 7548 Write 6 Done New Done false 6550 Acked 7549 Write 3 Done New Done false 6551 Acked 7550 Read 1 Done New Done false 6552 Acked 7551 Write 8 Done New Done false 6553 Acked 7552 Write 9 Done New Done false 6554 Acked 7553 Read 9 Done New Done false 6555 Acked 7554 Write 2 Done New Done false 6556 Acked 7555 Flush 0 Done New Done false 6557 Acked 7556 Read 4 Done New Done false 6558 Acked 7557 Write 3 Done New Done false 6559 Acked 7558 Read 4 Done New Done false 6560 Acked 7559 Read 9 Done New Done false 6561 Acked 7560 Write 6 Done New Done false 6562 Acked 7561 Read 9 Done New Done false 6563 Acked 7562 Write 2 Done New Done false 6564 Acked 7563 Read 2 Done New Done false 6565 Acked 7564 Read 3 Done New Done false 6566 Acked 7565 Write 8 Done New Done false 6567 Acked 7566 Read 10 Done New Done false 6568 Acked 7567 Read 8 Done New Done false 6569 Acked 7568 Write 6 Done New Done false 6570 Acked 7569 Read 2 Done New Done false 6571 Acked 7570 Read 4 Done New Done false 6572 Acked 7571 Flush 0 Done New Done false 6573 Acked 7572 Write 2 Done New Done false 6574 Acked 7573 Read 3 Done New Done false 6575 Acked 7574 Flush 0 Done New Done false 6576 Acked 7575 Write 4 Done New Done false 6577 Acked 7576 Read 5 Done New Done false 6578 Acked 7577 Read 6 Done New Done false 6579 Acked 7578 Read 7 Done New Done false 6580 Acked 7579 Write 7 Done New Done false 6581 Acked 7580 Write 3 Done New Done false 6582 Acked 7581 Read 10 Done New Done false 6583 Acked 7582 Write 4 Done New Done false 6584 Acked 7583 Read 6 Done New Done false 6585 Acked 7584 Read 4 Done New Done false 6586 Acked 7585 Flush 0 Done New Done false 6587 Acked 7586 Read 3 Done New Done false 6588 Acked 7587 Write 9 Done New Done false 6589 Acked 7588 Read 7 Done New Done false 6590 Acked 7589 Write 4 Done New Done false 6591 Acked 7590 Read 3 Done New Done false 6592 Acked 7591 Read 7 Done New Done false 6593 Acked 7592 Write 8 Done New Done false 6594 Acked 7593 Write 9 Done New Done false 6595 Acked 7594 Read 8 Done New Done false 6596 Acked 7595 Write 7 Done New Done false 6597 Acked 7596 Flush 0 Done New Done false 6598 Acked 7597 Write 6 Done New Done false 6599 Acked 7598 Read 10 Done New Done false 6600 Acked 7599 Read 2 Done New Done false 6601 Acked 7600 Flush 0 Done New Done false 6602 Acked 7601 Flush 0 Done New Done false 6603 Acked 7602 Write 9 Done New Done false 6604 Acked 7603 Read 4 Done New Done false 6605 Acked 7604 Write 8 Done New Done false 6606 Acked 7605 Read 6 Done New Done false 6607 Acked 7606 Write 6 Done New Done false 6608 Acked 7607 Write 3 Done New Done false 6609 Acked 7608 Read 10 Done New Done false 6610 Acked 7609 Read 6 Done New Done false 6611 Acked 7610 Read 9 Done New Done false 6612 Acked 7611 Write 9 Done New Done false 6613 Acked 7612 Read 10 Done New Done false 6614 Acked 7613 Read 8 Done New Done false 6615 Acked 7614 Flush 0 Done New Done false 6616 Acked 7615 Write 1 Done New Done false 6617 Acked 7616 Write 6 Done New Done false 6618 Acked 7617 Read 10 Done New Done false 6619 Acked 7618 Write 5 Done New Done false 6620 Acked 7619 Flush 0 Done New Done false 6621 Acked 7620 Read 9 Done New Done false 6622 Acked 7621 Read 7 Done New Done false 6623 Acked 7622 Read 9 Done New Done false 6624 Acked 7623 Flush 0 Done New Done false 6625 Acked 7624 Write 1 Done New Done false 6626 Acked 7625 Read 10 Done New Done false 6627 Acked 7626 Read 9 Done New Done false 6628 Acked 7627 Flush 0 Done New Done false 6629 Acked 7628 Read 6 Done New Done false 6630 Acked 7629 Write 3 Done New Done false 6631 Acked 7630 Write 10 Done New Done false 6632 Acked 7631 Read 8 Done New Done false 6633 Acked 7632 Read 3 Done New Done false 6634 Acked 7633 Write 8 Done New Done false 6635 Acked 7634 Write 4 Done New Done false 6636 Acked 7635 Flush 0 Done New Done false 6637 Acked 7636 Write 8 Done New Done false 6638 Acked 7637 Read 10 Done New Done false 6639 Acked 7638 Read 1 Done New Done false 6640 Acked 7639 Write 8 Done New Done false 6641 Acked 7640 Flush 0 Done New Done false 6642 Acked 7641 Read 10 Done New Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 300 0 300 Sent 0 0 0 0 Done 300 0 300 600 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7640 7341 7640 Downstairs last five completed: 7341 7340 7339 7338 7337 Upstairs last five completed: 6642 6641 6640 6639 6638 CLIENT: Up:0 ds:300 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:51.658248123Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:51.658316565Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:51.658344877Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:51.658369287Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:51.65839568Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:51.658416267Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:51.658438721Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:51.658668841Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:52.659532372Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:52.659599715Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:52.659627899Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:52.65965427Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:52.659674767Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:52.659707638Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:52.659961334Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:53.660971979Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:53.661039046Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:53.661067124Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:53.661093635Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:53.661114089Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:53.66114441Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:53.661384911Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:54.66222672Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:54.662293618Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:54.662322297Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:54.662348795Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:54.662369247Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:54.662399474Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:54.662641138Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Offline Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 6343 Acked 7342 Read 9 Done New Done true 6344 Acked 7343 Read 1 Done New Done true 6345 Acked 7344 Flush 0 Done New Done true 6346 Acked 7345 Write 9 Done New Done true 6347 Acked 7346 Write 3 Done New Done true 6348 Acked 7347 Write 8 Done New Done true 6349 Acked 7348 Write 5 Done New Done true 6350 Acked 7349 Write 2 Done New Done true 6351 Acked 7350 Write 5 Done New Done true 6352 Acked 7351 Read 8 Done New Done true 6353 Acked 7352 Write 5 Done New Done true 6354 Acked 7353 Read 9 Done New Done true 6355 Acked 7354 Flush 0 Done New Done true 6356 Acked 7355 Read 3 Done New Done true 6357 Acked 7356 Write 1 Done New Done true 6358 Acked 7357 Read 5 Done New Done true 6359 Acked 7358 Write 3 Done New Done true 6360 Acked 7359 Read 10 Done New Done true 6361 Acked 7360 Read 10 Done New Done true 6362 Acked 7361 Read 5 Done New Done true 6363 Acked 7362 Write 5 Done New Done true 6364 Acked 7363 Read 10 Done New Done true 6365 Acked 7364 Read 1 Done New Done true 6366 Acked 7365 Read 6 Done New Done true 6367 Acked 7366 Read 1 Done New Done true 6368 Acked 7367 Flush 0 Done New Done true 6369 Acked 7368 Flush 0 Done New Done true 6370 Acked 7369 Write 9 Done New Done true 6371 Acked 7370 Write 1 Done New Done true 6372 Acked 7371 Flush 0 Done New Done true 6373 Acked 7372 Read 9 Done New Done true 6374 Acked 7373 Read 7 Done New Done true 6375 Acked 7374 Write 5 Done New Done true 6376 Acked 7375 Write 4 Done New Done true 6377 Acked 7376 Write 7 Done New Done true 6378 Acked 7377 Read 3 Done New Done true 6379 Acked 7378 Write 9 Done New Done true 6380 Acked 7379 Read 5 Done New Done true 6381 Acked 7380 Write 7 Done New Done true 6382 Acked 7381 Read 8 Done New Done true 6383 Acked 7382 Read 3 Done New Done true 6384 Acked 7383 Write 3 Done New Done true 6385 Acked 7384 Write 2 Done New Done true 6386 Acked 7385 Read 10 Done New Done true 6387 Acked 7386 Read 1 Done New Done true 6388 Acked 7387 Read 2 Done New Done true 6389 Acked 7388 Write 4 Done New Done true 6390 Acked 7389 Write 8 Done New Done true 6391 Acked 7390 Flush 0 Done New Done true 6392 Acked 7391 Write 6 Done New Done true 6393 Acked 7392 Read 7 Done New Done true 6394 Acked 7393 Read 7 Done New Done true 6395 Acked 7394 Read 9 Done New Done true 6396 Acked 7395 Read 4 Done New Done true 6397 Acked 7396 Write 1 Done New Done true 6398 Acked 7397 Read 2 Done New Done true 6399 Acked 7398 Read 10 Done New Done true 6400 Acked 7399 Write 1 Done New Done true 6401 Acked 7400 Write 4 Done New Done true 6402 Acked 7401 Read 2 Done New Done true 6403 Acked 7402 Write 10 Done New Done true 6404 Acked 7403 Read 8 Done New Done true 6405 Acked 7404 Read 2 Done New Done true 6406 Acked 7405 Read 6 Done New Done true 6407 Acked 7406 Read 6 Done New Done true 6408 Acked 7407 Flush 0 Done New Done true 6409 Acked 7408 Flush 0 Done New Done true 6410 Acked 7409 Write 6 Done New Done true 6411 Acked 7410 Read 2 Done New Done true 6412 Acked 7411 Write 5 Done New Done true 6413 Acked 7412 Read 5 Done New Done true 6414 Acked 7413 Read 3 Done New Done true 6415 Acked 7414 Write 8 Done New Done true 6416 Acked 7415 Read 5 Done New Done true 6417 Acked 7416 Write 5 Done New Done true 6418 Acked 7417 Read 6 Done New Done true 6419 Acked 7418 Write 7 Done New Done true 6420 Acked 7419 Write 3 Done New Done true 6421 Acked 7420 Write 1 Done New Done true 6422 Acked 7421 Read 6 Done New Done true 6423 Acked 7422 Write 3 Done New Done true 6424 Acked 7423 Read 4 Done New Done true 6425 Acked 7424 Write 5 Done New Done true 6426 Acked 7425 Write 3 Done New Done true 6427 Acked 7426 Write 6 Done New Done true 6428 Acked 7427 Flush 0 Done New Done true 6429 Acked 7428 Flush 0 Done New Done true 6430 Acked 7429 Write 2 Done New Done true 6431 Acked 7430 Read 6 Done New Done true 6432 Acked 7431 Flush 0 Done New Done true 6433 Acked 7432 Write 1 Done New Done true 6434 Acked 7433 Write 6 Done New Done true 6435 Acked 7434 Read 4 Done New Done true 6436 Acked 7435 Read 9 Done New Done true 6437 Acked 7436 Read 9 Done New Done true 6438 Acked 7437 Read 7 Done New Done true 6439 Acked 7438 Read 6 Done New Done true 6440 Acked 7439 Flush 0 Done New Done true 6441 Acked 7440 Flush 0 Done New Done true 6442 Acked 7441 Read 8 Done New Done true 6443 Acked 7442 Flush 0 Done New Done true 6444 Acked 7443 Read 6 Done New Done true 6445 Acked 7444 Read 2 Done New Done true 6446 Acked 7445 Read 7 Done New Done true 6447 Acked 7446 Read 5 Done New Done true 6448 Acked 7447 Write 4 Done New Done true 6449 Acked 7448 Read 10 Done New Done true 6450 Acked 7449 Read 6 Done New Done true 6451 Acked 7450 Write 6 Done New Done true 6452 Acked 7451 Write 2 Done New Done true 6453 Acked 7452 Read 4 Done New Done true 6454 Acked 7453 Read 9 Done New Done true 6455 Acked 7454 Read 10 Done New Done true 6456 Acked 7455 Write 2 Done New Done true 6457 Acked 7456 Write 9 Done New Done true 6458 Acked 7457 Read 2 Done New Done true 6459 Acked 7458 Write 5 Done New Done true 6460 Acked 7459 Write 4 Done New Done true 6461 Acked 7460 Flush 0 Done New Done true 6462 Acked 7461 Write 4 Done New Done true 6463 Acked 7462 Read 10 Done New Done true 6464 Acked 7463 Flush 0 Done New Done true 6465 Acked 7464 Write 6 Done New Done true 6466 Acked 7465 Read 4 Done New Done true 6467 Acked 7466 Read 10 Done New Done true 6468 Acked 7467 Read 7 Done New Done true 6469 Acked 7468 Read 8 Done New Done true 6470 Acked 7469 Read 5 Done New Done true 6471 Acked 7470 Write 10 Done New Done true 6472 Acked 7471 Read 5 Done New Done true 6473 Acked 7472 Read 1 Done New Done true 6474 Acked 7473 Write 9 Done New Done true 6475 Acked 7474 Read 3 Done New Done true 6476 Acked 7475 Read 9 Done New Done true 6477 Acked 7476 Read 9 Done New Done true 6478 Acked 7477 Read 8 Done New Done true 6479 Acked 7478 Flush 0 Done New Done true 6480 Acked 7479 Read 2 Done New Done true 6481 Acked 7480 Read 9 Done New Done true 6482 Acked 7481 Write 5 Done New Done true 6483 Acked 7482 Write 4 Done New Done true 6484 Acked 7483 Read 6 Done New Done true 6485 Acked 7484 Read 2 Done New Done true 6486 Acked 7485 Read 1 Done New Done true 6487 Acked 7486 Write 6 Done New Done true 6488 Acked 7487 Read 9 Done New Done true 6489 Acked 7488 Write 1 Done New Done true 6490 Acked 7489 Read 8 Done New Done true 6491 Acked 7490 Write 2 Done New Done true 6492 Acked 7491 Write 9 Done New Done true 6493 Acked 7492 Read 9 Done New Done true 6494 Acked 7493 Write 9 Done New Done true 6495 Acked 7494 Read 4 Done New Done true 6496 Acked 7495 Read 4 Done New Done true 6497 Acked 7496 Read 4 Done New Done true 6498 Acked 7497 Read 4 Done New Done true 6499 Acked 7498 Write 7 Done New Done true 6500 Acked 7499 Write 4 Done New Done true 6501 Acked 7500 Read 2 Done New Done true 6502 Acked 7501 Write 8 Done New Done true 6503 Acked 7502 Read 7 Done New Done true 6504 Acked 7503 Read 6 Done New Done true 6505 Acked 7504 Write 8 Done New Done true 6506 Acked 7505 Read 9 Done New Done true 6507 Acked 7506 Read 6 Done New Done true 6508 Acked 7507 Read 9 Done New Done true 6509 Acked 7508 Flush 0 Done New Done true 6510 Acked 7509 Write 3 Done New Done true 6511 Acked 7510 Write 7 Done New Done true 6512 Acked 7511 Read 6 Done New Done true 6513 Acked 7512 Flush 0 Done New Done true 6514 Acked 7513 Read 5 Done New Done true 6515 Acked 7514 Write 1 Done New Done true 6516 Acked 7515 Write 4 Done New Done true 6517 Acked 7516 Read 6 Done New Done true 6518 Acked 7517 Write 2 Done New Done true 6519 Acked 7518 Write 4 Done New Done true 6520 Acked 7519 Flush 0 Done New Done true 6521 Acked 7520 Write 3 Done New Done true 6522 Acked 7521 Write 4 Done New Done true 6523 Acked 7522 Flush 0 Done New Done true 6524 Acked 7523 Read 8 Done New Done true 6525 Acked 7524 Read 10 Done New Done true 6526 Acked 7525 Read 4 Done New Done true 6527 Acked 7526 Flush 0 Done New Done true 6528 Acked 7527 Write 3 Done New Done true 6529 Acked 7528 Read 10 Done New Done true 6530 Acked 7529 Read 7 Done New Done true 6531 Acked 7530 Write 10 Done New Done true 6532 Acked 7531 Write 10 Done New Done true 6533 Acked 7532 Write 4 Done New Done true 6534 Acked 7533 Read 10 Done New Done true 6535 Acked 7534 Write 6 Done New Done true 6536 Acked 7535 Write 2 Done New Done true 6537 Acked 7536 Read 9 Done New Done true 6538 Acked 7537 Read 8 Done New Done true 6539 Acked 7538 Read 9 Done New Done true 6540 Acked 7539 Write 5 Done New Done true 6541 Acked 7540 Write 1 Done New Done true 6542 Acked 7541 Write 7 Done New Done true 6543 Acked 7542 Read 1 Done New Done true 6544 Acked 7543 Write 4 Done New Done true 6545 Acked 7544 Read 6 Done New Done true 6546 Acked 7545 Write 1 Done New Done true 6547 Acked 7546 Write 3 Done New Done true 6548 Acked 7547 Read 2 Done New Done true 6549 Acked 7548 Write 6 Done New Done true 6550 Acked 7549 Write 3 Done New Done true 6551 Acked 7550 Read 1 Done New Done true 6552 Acked 7551 Write 8 Done New Done true 6553 Acked 7552 Write 9 Done New Done true 6554 Acked 7553 Read 9 Done New Done true 6555 Acked 7554 Write 2 Done New Done true 6556 Acked 7555 Flush 0 Done New Done true 6557 Acked 7556 Read 4 Done New Done true 6558 Acked 7557 Write 3 Done New Done true 6559 Acked 7558 Read 4 Done New Done true 6560 Acked 7559 Read 9 Done New Done true 6561 Acked 7560 Write 6 Done New Done true 6562 Acked 7561 Read 9 Done New Done true 6563 Acked 7562 Write 2 Done New Done true 6564 Acked 7563 Read 2 Done New Done true 6565 Acked 7564 Read 3 Done New Done true 6566 Acked 7565 Write 8 Done New Done true 6567 Acked 7566 Read 10 Done New Done true 6568 Acked 7567 Read 8 Done New Done true 6569 Acked 7568 Write 6 Done New Done true 6570 Acked 7569 Read 2 Done New Done true 6571 Acked 7570 Read 4 Done New Done true 6572 Acked 7571 Flush 0 Done New Done true 6573 Acked 7572 Write 2 Done New Done true 6574 Acked 7573 Read 3 Done New Done true 6575 Acked 7574 Flush 0 Done New Done true 6576 Acked 7575 Write 4 Done New Done true 6577 Acked 7576 Read 5 Done New Done true 6578 Acked 7577 Read 6 Done New Done true 6579 Acked 7578 Read 7 Done New Done true 6580 Acked 7579 Write 7 Done New Done true 6581 Acked 7580 Write 3 Done New Done true 6582 Acked 7581 Read 10 Done New Done true 6583 Acked 7582 Write 4 Done New Done true 6584 Acked 7583 Read 6 Done New Done true 6585 Acked 7584 Read 4 Done New Done true 6586 Acked 7585 Flush 0 Done New Done true 6587 Acked 7586 Read 3 Done New Done true 6588 Acked 7587 Write 9 Done New Done true 6589 Acked 7588 Read 7 Done New Done true 6590 Acked 7589 Write 4 Done New Done true 6591 Acked 7590 Read 3 Done New Done true 6592 Acked 7591 Read 7 Done New Done true 6593 Acked 7592 Write 8 Done New Done true 6594 Acked 7593 Write 9 Done New Done true 6595 Acked 7594 Read 8 Done New Done true 6596 Acked 7595 Write 7 Done New Done true 6597 Acked 7596 Flush 0 Done New Done true 6598 Acked 7597 Write 6 Done New Done true 6599 Acked 7598 Read 10 Done New Done true 6600 Acked 7599 Read 2 Done New Done true 6601 Acked 7600 Flush 0 Done New Done true 6602 Acked 7601 Flush 0 Done New Done true 6603 Acked 7602 Write 9 Done New Done true 6604 Acked 7603 Read 4 Done New Done true 6605 Acked 7604 Write 8 Done New Done true 6606 Acked 7605 Read 6 Done New Done true 6607 Acked 7606 Write 6 Done New Done true 6608 Acked 7607 Write 3 Done New Done true 6609 Acked 7608 Read 10 Done New Done true 6610 Acked 7609 Read 6 Done New Done true 6611 Acked 7610 Read 9 Done New Done true 6612 Acked 7611 Write 9 Done New Done true 6613 Acked 7612 Read 10 Done New Done true 6614 Acked 7613 Read 8 Done New Done true 6615 Acked 7614 Flush 0 Done New Done true 6616 Acked 7615 Write 1 Done New Done true 6617 Acked 7616 Write 6 Done New Done true 6618 Acked 7617 Read 10 Done New Done true 6619 Acked 7618 Write 5 Done New Done true 6620 Acked 7619 Flush 0 Done New Done true 6621 Acked 7620 Read 9 Done New Done true 6622 Acked 7621 Read 7 Done New Done true 6623 Acked 7622 Read 9 Done New Done true 6624 Acked 7623 Flush 0 Done New Done true 6625 Acked 7624 Write 1 Done New Done true 6626 Acked 7625 Read 10 Done New Done true 6627 Acked 7626 Read 9 Done New Done true 6628 Acked 7627 Flush 0 Done New Done true 6629 Acked 7628 Read 6 Done New Done true 6630 Acked 7629 Write 3 Done New Done true 6631 Acked 7630 Write 10 Done New Done true 6632 Acked 7631 Read 8 Done New Done true 6633 Acked 7632 Read 3 Done New Done true 6634 Acked 7633 Write 8 Done New Done true 6635 Acked 7634 Write 4 Done New Done true 6636 Acked 7635 Flush 0 Done New Done true 6637 Acked 7636 Write 8 Done New Done true 6638 Acked 7637 Read 10 Done New Done true 6639 Acked 7638 Read 1 Done New Done true 6640 Acked 7639 Write 8 Done New Done true 6641 Acked 7640 Flush 0 Done New Done true 6642 Acked 7641 Read 10 Done New Done true 6643 Acked 7642 Flush 0 Done New Done true STATES DS:0 DS:1 DS:2 TOTAL New 0 301 0 301 Sent 0 0 0 0 Done 301 0 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7642 7341 7642 Downstairs last five completed: 7341 7340 7339 7338 7337 Upstairs last five completed: 6643 6642 6641 6640 6639 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:55.663791916Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:55.66385666Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:55.663884994Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:55.663911448Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:55.663931825Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:55.663963751Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:55.66423324Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8820] got start action [8820] I am going to restart Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8820.txt" [8820][1] reports Starting {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:56.66607431Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:56.666141957Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:56.666177535Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:56.666202414Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:56.666228643Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:56.666248754Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] client re-new 301 jobs since flush 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:56.666271171Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:56.666552657Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1207) [8820][1] reports Running {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.667545111Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.667686181Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.669661304Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.669698563Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.669722497Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 7341","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.670005613Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.670040395Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.670069734Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.670092826Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:57.670114785Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"1","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7642 7641 7640 7639 7638 Upstairs last five completed: 6643 6642 6641 6640 6639 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:34:59.288 INFO request completed, latency_us: 454, response_code: 204, uri: /stop/cid/0, method: POST, req_id: f2253523-5e57-4669-825d-5e992b575885, remote_addr:[8810] Got stop action so:false kr:false 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:34:59.289 INFO request completed, latency_us: 327, response_code: 200, uri: /state/cid/0, method: GET, req_id: 86813b03-c95d-4ff1-8481-35e8acade53d, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:59.290294802Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:59.290374651Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:34:59.290421531Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:59.290458742Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:59.290502334Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:34:59.290553829Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:00.291681095Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:00.29176243Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:00.291846055Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:00.291877997Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:00.291898594Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:00.291920979Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:00.291943231Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:01.293836189Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:01.293949861Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:01.294006522Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:01.29406014Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:01.29410529Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:01.29415252Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:01.294198907Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:02.295901454Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:02.29597563Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:02.296015226Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:02.296040925Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:02.296067314Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:02.296087791Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:02.29611015Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:02.296211187Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:35:03.292 INFO request completed, latency_us: 260, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2cc32233-6269-4de4-a4ad-cca579386324, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 1085 len 4096 data: 5 002/300 Read block 1893 len 16384 003/300 Write block 3192 len 4096 data: 6 004/300 Read block 1981 len 4096 005/300 Write block 1881 len 36864 data: 5 5 5 4 6 7 5 6 5 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:03.29811494Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:03.298210616Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:03.298263524Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:03.298295698Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:03.298316404Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 006/300{ Write block 635 len 16384 data:" 6msg 3":" 3 4 [0] client re-new 5 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:03.298404111Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:03.298432622Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 007/300 Read block 1461 len 40960 008/300 Read block 2009 len 12288 009/300 Write block 1086 len 20480 data: 4 3 3 6 6 010/300 Flush 011/300 Flush 012/300 Flush 013/300 Read block 4355 len 4096 014/300 Read block 350 len 32768 015/300 Read block 458 len 4096 016/300 Flush 017/300 Read block 3770 len 12288 018/300 Read block 4144 len 20480 019/300 Flush 020/300 Read block 4887 len 4096 021/300 Write block 4464 len 28672 data: 2 2 2 2 2 2 3 022/300 Write block 2865 len 20480 data: 5 5 6 6 5 023/300 Read block 921 len 36864 024/300 Write block 3768 len 32768 data: 8 6 6 6 4 2 4 4 025/300 Read block 1571 len 32768 026/300 Read block 2972 len 20480 027/300 Read block 1084 len 20480 028/300 Write block 1814 len 8192 data: 2 2 029/300 Read block 821 len 24576 030/300 Write block 3883 len 28672 data: 8 6 7 8 7 6 4 031/300 Read block 2646 len 8192 032/300 Read block 1696 len 24576 033/300 Read block 1476 len 12288 034/300 Read block 4427 len 12288 035/300 Read block 379 len 4096 036/300 Read block 728 len 8192 037/300 Flush 038/300 Write block 4956 len 28672 data: 4 6 6 4 4 3 5 039/300 Write block 176 len 20480 data: 3 3 2 2 2 040/300 Read block 3613 len 16384 041/300 Write block 1065 len 36864 data: 4 5 5 5 6 8 7 10 10 042/300 Read block 1861 len 32768 043/300 Read block 4046 len 16384 044/300 Read block 644 len 28672 045/300 Flush 046/300 Read block 1986 len 4096 047/300 Write block 2221 len 24576 data: 2 2 2 2 3 4 048/300 Flush 049/300 Flush 050/300 Read block 4249 len 4096 051/300 Write block 622 len 8192 data: 3 2 052/300 Read block 1144 len 36864 053/300 Read block 2969 len 24576 054/300 Read block 3679 len 24576 055/300 Write block 3727 len 36864 data: 5 5 5 5 5 2 2 3 3 056/300 Write block 1897 len 24576 data: 4 3 3 3 5 5 057/300 Read block 4820 len 20480 058/300 Read block 2249 len 28672 059/300 Read block 1349 len 32768 060/300 Write block 1294 len 16384 data: 5 6 8 8 061/300 Write block 3447 len 32768 data: 3 3 3 2 4 5 7 8 062/300 Flush 063/300 Flush 064/300 Write block 4572 len 24576 data: 5 3 4 6 7 7 065/300 Read block 4876 len 4096 066/300 Write block 956 len 40960 data: 3 3 3 3 3 3 4 4 4 4 067/300 Flush 068/300 Read block 3960 len 16384 069/300 Read block 122 len 8192 070/300 Write block 2738 len 36864 data: 4 4 4 4 4 3 3 2 4 071/300 Flush 072/300 Write block 1502 len 24576 data: 4 6 7 6 7 6 073/300 Read block 1096 len 36864 074/300 Write block 4485 len 20480 data: 4 3 3 3 4 075/300 Read block 4571 len 36864 076/300 Write block 2473 len 20480 data: 4 4 5 5 5 077/300 Write block 3952 len 32768 data: 3 4 5 5 4 4 6 7 078/300 Write block 3297 len 24576 data: 3 3 3 4 4 5 079/300 Read block 3037 len 24576 080/300 Read block 1466 len 12288 081/300 Write block 1274 len 40960 data: 4 5 6 6 4 3 3 4 4 5 082/300 Read block 4038 len 40960 083/300 Read block 235 len 20480 084/300 Flush 085/300 Read block 734 len 12288 086/300 Read block 4405 len 12288 087/300 Flush 088/300 Read block 2473 len 32768 089/300 Write block 229 len 8192 data: 4 3 090/300 Read block 4805 len 36864 091/300 Write block 2798 len 24576 data: 4 7 7 6 6 6 092/300 Write block 1882 len 28672 data: 6 6 5 7 8 6 7 093/300 Write block 2686 len 28672 data: 5 6 8 7 8 8 8 094/300 Write block 3792 len 12288 data: 8 6 6 095/300 Write block 1584 len 4096 data: 4 096/300 Read block 2721 len 40960 097/300 Write block 395 len 20480 data: 9 6 4 6 7 098/300 Flush 099/300 Flush 100/300 Write block 1688 len 20480 data: 9 6 6 6 5 101/300 Write block 1093 len 12288 data: 6 7 6 102/300 Read block 2169 len 40960 103/300 Read block 3208 len 32768 104/300 Read block 3077 len 20480 105/300 Read block 4251 len 20480 106/300 Write block 1463 len 4096 data: 6 107/300 Flush 108/300 Read block 3357 len 12288 109/300 Read block 466 len 28672 110/300 Write block 4382 len 12288 data: 3 5 4 111/300 Flush 112/300 Write block 4656 len 8192 data: 6 6 113/300 Write block 2226 len 16384 data: 5 5 5 5 114/300 Write block 3474 len 20480 data: 5 4 4 5 5 115/300 Read block 1730 len 4096 116/300 Read block 4202 len 16384 117/300 Write block 3028 len 4096 data: 5 118/300 Write block 2362 len 36864 data: 4 6 5 6 6 4 3 3 3 119/300 Read block 4478 len 20480 120/300 Read block 800 len 8192 121/300 Write block 1117 len 8192 data: 4 4 122/300 Flush 123/300 Read block 2019 len 8192 124/300 Write block 1910 len 40960 data: 3 5 6 5 4 4 4 4 5 4 125/300 Flush 126/300 Read block 333 len 36864 127/300 Read block 2765 len 32768 128/300 Read block 4316 len 20480 129/300 Read block 2821 len 12288 130/300 Read block 753 len 24576 131/300 Read block 16 len 28672 132/300 Write block 1031 len 4096 data: 3 133/300 Write block 1323 len 28672 data: 6 5 5 5 6 5 5 134/300 Write block 1129 len 24576 data: 6 5 5 6 6 8 135/300 Write block 598 len 24576 data: 5 5 5 5 5 3 136/300 Write block 1812 len 28672 data: 2 2 3 3 3 3 4 137/300 Read block 711 len 8192 138/300 Read block 3817 len 24576 139/300 Read block 4288 len 8192 140/300 Write block 4754 len 20480 data: 4 4 4 6 5 141/300 Read block 3312 len 16384 142/300 Write block 4348 len 4096 data: 4 143/300 Write block 4417 len 12288 data: 3 5 6 144/300 Write block 4931 len 4096 data: 10 145/300 Write block 2223 len 16384 data: 3 3 4 6 146/300 Read block 816 len 16384 147/300 Write block 3813 len 24576 data: 4 4 4 4 5 4 148/300 Write block 2006 len 12288 data: 5 7 9 149/300 Write block 1838 len 16384 data: 3 3 3 3 150/300 Read block 2845 len 20480 151/300 Read block 1906 len 12288 152/300 Read block 2472 len 16384 153/300 Write block 4749 len 36864 data: 5 5 5 4 5 5 5 5 7 154/300 Write block 99 len 4096 data: 7 155/300 Write block 1836 len 32768 data: 3 3 4 4 4 4 3 4 156/300 Read block 847 len 36864 157/300 Write block 2902 len 20480 data: 4 5 4 4 6 158/300 Write block 2117 len 20480 data: 4 4 5 4 6 159/300 Write block 4743 len 12288 data: 5 4 4 160/300 Read block 4843 len 40960 161/300 Read block 3368 len 4096 162/300 Read block 1856 len 24576 163/300 Write block 3813 len 36864 data: 5 5 5 5 6 5 4 4 4 164/300 Read block 1287 len 40960 165/300 Write block 2714 len 4096 data: 5 166/300 Read block 4089 len 12288 167/300 Read block 1005 len 24576 168/300 Write block 1329 len 24576 data: 6 5 5 6 7 6 169/300 Read block 2389 len 40960 170/300 Read block 4200 len 20480 171/300 Read block 1776 len 12288 172/300 Write block 3685 len 40960 data: 4 2 2 2 3 3 4 4 4 4 173/300 Read block 1363 len 36864 174/300 Flush 175/300 Read block 2547 len 16384 176/300 Read block 2285 len 32768 177/300 Read block 4785 len 36864 178/300 Read block 1196 len 32768 179/300 Write block 2030 len 12288 data: 7 8 9 180/300 Read block 3672 len 40960 181/300 Flush 182/300 Read block 279 len 16384 183/300 Read block 2197 len 24576 184/300 Read block 2730 len 12288 185/300 Read block 3593 len 24576 186/300 Read block 642 len 20480 187/300 Write block 4050 len 36864 data: 6 5 5 4 4 4 3 2 3 188/300 Write block 1443 len 8192 data: 4 4 189/300 Write block 1050 len 12288 data: 4 4 5 190/300 Read block 2869 len 40960 191/300 Flush 192/300 Write block 3960 len 4096 data: 8 193/300 Read block 2263 len 4096 194/300 Read block 845 len 24576 195/300 Read block 790 len 28672 196/300 Write block 2476 len 8192 data: 6 6 197/300 Flush 198/300 Write block 1190 len 16384 data: 5 5 5 4 199/300 Read block 1859 len 32768 200/300 Write block 2923 len 8192 data: 4 4 201/300 Write block 2042 len 12288 data: 3 4 4 202/300 Write block 297 len 20480 data: 7 5 3 3 3 203/300 Write block 146 len 32768 data: 6 5 6 6 5 4 4 4 204/300 Write block 1581 len 20480 data: 5 5 5 5 8 205/300 Write block 1394 len 12288 data: 7 7 7 206/300 Write block 1751 len 28672 data: 4 4 5 6 8 8 9 207/300 Write block 4260 len 28672 data: 9 8 7 5 5 7 9 208/300 Read block 2848 len 32768 209/300 Read block 2477 len 36864 210/300 Flush 211/300 Flush 212/300 Read block 1074 len 16384 213/300 Write block 1115 len 40960 data: 7 5 5 5 4 4 5 4 3 3 214/300 Read block 3347 len 8192 215/300 Write block 1960 len 8192 data: 3 3 216/300 Write block 4426 len 32768 data: 5 4 4 4 3 3 5 5 217/300 Flush 218/300 Read block 2111 len 40960 219/300 Write block 3209 len 40960 data: 2 2 2 4 5 5 5 3 3 3 220/300 Read block 619 len 8192 221/300 Flush 222/300 Read block 496 len 40960 223/300 Read block 552 len 16384 224/300 Write block 1179 len 24576 data: 4 4 5 4 3 3 225/300 Write block 2053 len 28672 data: 6 6 6 7 7 5 5 226/300 Flush 227/300 Write block 3240 len 20480 data: 8 8 7 7 6 228/300 Read block 2986 len 36864 229/300 Write block 3740 len 36864 data: 6 3 3 3 3 4 4 3 4 230/300 Read block 166 len 16384 231/300 Write block 2675 len 4096 data: 3 232/300 Flush 233/300 Read block 1509 len 28672 234/300 Write block 3725 len 36864 data: 6 6 6 6 6 6 6 3 3 235/300 Read block 3718 len 24576 236/300 Write block 3424 len 12288 data: 6 6 5 237/300 Read block 3537 len 32768 238/300 Read block 3191 len 28672 239/300 Write block 1864 len 8192 data: 3 3 240/300 Read block 1607 len 16384 241/300 Write block 2349 len 24576 data: 4 5 3 3 3 3 242/300 Write block 2532 len 36864 data: 6 6 5 5 5 4 4 5 6 243/300 Write block 509 len 32768 data: 3 2 3 4 5 6 6 8 244/300 Read block 3193 len 36864 245/300 Write block 3215 len 32768 data: 6 4 4 4 3 5 4 4 246/300 Write block 872 len 32768 data: 4 4 3 2 2 2 4 3 247/300 Write block 2251 len 40960 data: 3 3 3 3 5 7 6 7 8 7 248/300 Write block 540 len 36864 data: 6 6 5 4 4 4 5 6 6 249/300 Read block 462 len 4096 250/300 Read block 1660 len 36864 251/300 Read block 2965 len 32768 252/300 Write block 349 len 40960 data: 4 4 4 4 5 5 4 3 3 3 253/300 Read block 1809 len 36864 254/300 Read block 3033 len 12288 255/300 Read block 3229 len 4096 256/300 Read block 747 len 8192 257/300 Read block 738 len 24576 258/300 Flush 259/300 Write block 3988 len 8192 data: 8 8 260/300 Flush 261/300 Flush 262/300 Write block 2306 len 4096 data: 7 263/300 Read block 4910 len 24576 264/300 Write block 1672 len 8192 data: 7 7 265/300 Write block 2382 len 4096 data: 2 266/300 Read block 2950 len 20480 267/300 Read block 237 len 12288 268/300 Write block 515 len 16384 data: 7 9 6 7 269/300 Write block 3564 len 28672 data: 2 3 3 3 3 4 3 270/300 Write block 2682 len 20480 data: 5 4 4 6 6 271/300 Read block 555 len 28672 272/300 Read block 4845 len 28672 273/300 Read block 4114 len 8192 274/300 Read block 3117 len 36864 275/300 Read block 1117 len 8192 276/300 Write block 3358 len 28672 data: 6 8 8 6 6 5 4 277/300 Read block 4419 len 8192 278/300 Read block 2387 len 28672 279/300 Read block 2820 len 40960 280/300 Write block 3888 len 28672 data: 7 5 4 4 5 5 5 281/300 Flush 282/300 Read block 275 len 8192 283/300 Read block 106 len 24576 284/300 Flush 285/300 Read block 4220 len 32768 286/300 Write block 47 len 32768 data: 5 6 6 5 4 4 4 4 287/300 Flush 288/300 Write block 657 len 12288 data: 9 11 12 289/300 Write block 1465 len 32768 data: 5 4 5 5 5 4 4 3 290/300 Write block 1650 len 24576 data: 3 4 4 5 4 6 291/300 Write block 4306 len 12288 data: 3 6 5 292/300 Read block 2900 len 8192 293/300 Read block 464 len 20480 294/300 Read block 1562 len 12288 295/300 Write block 2199 len 28672 data: 6 6 6 4 3 3 3 296/300 Read block 4696 len 32768 297/300 Read block 754 len 36864 298/300 Write block 558 len 32768 data: 3 2 2 2 2 3 4 4 299/300 Write block 3477 len 28672 data: 6 6 5 4 5 6 6 300/300 Read block 4448 len 4096 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:35:03.872 INFO request completed, latency_us: 295, response_code: 204, uri: /start/cid/0, method: POST, req_id: e471dca1-4757-420d-9550-70a96e74597e, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:1 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 6644 Acked 7643 Write 1 New Done Done true 6645 Acked 7644 Read 4 New Done Done true 6646 Acked 7645 Write 1 New Done Done true 6647 Acked 7646 Read 1 New Done Done true 6648 Acked 7647 Write 9 New Done Done true 6649 Acked 7648 Write 4 New Done Done false 6650 Acked 7649 Read 10 New Done Done false 6651 Acked 7650 Read 3 New Done Done false 6652 Acked 7651 Write 5 New Done Done false 6653 Acked 7652 Flush 0 New Done Done false 6654 Acked 7653 Flush 0 New Done Done false 6655 Acked 7654 Flush 0 New Done Done false 6656 Acked 7655 Read 1 New Done Done false 6657 Acked 7656 Read 8 New Done Done false 6658 Acked 7657 Read 1 New Done Done false 6659 Acked 7658 Flush 0 New Done Done false 6660 Acked 7659 Read 3 New Done Done false 6661 Acked 7660 Read 5 New Done Done false 6662 Acked 7661 Flush 0 New Done Done false 6663 Acked 7662 Read 1 New Done Done false 6664 Acked 7663 Write 7 New Done Done false 6665 Acked 7664 Write 5 New Done Done false 6666 Acked 7665 Read 9 New Done Done false 6667 Acked 7666 Write 8 New Done Done false 6668 Acked 7667 Read 8 New Done Done false 6669 Acked 7668 Read 5 New Done Done false 6670 Acked 7669 Read 5 New Done Done false 6671 Acked 7670 Write 2 New Done Done false 6672 Acked 7671 Read 6 New Done Done false 6673 Acked 7672 Write 7 New Done Done false 6674 Acked 7673 Read 2 New Done Done false 6675 Acked 7674 Read 6 New Done Done false 6676 Acked 7675 Read 3 New Done Done false 6677 Acked 7676 Read 3 New Done Done false 6678 Acked 7677 Read 1 New Done Done false 6679 Acked 7678 Read 2 New Done Done false 6680 Acked 7679 Flush 0 New Done Done false 6681 Acked 7680 Write 7 New Done Done false 6682 Acked 7681 Write 5 New Done Done false 6683 Acked 7682 Read 4 New Done Done false 6684 Acked 7683 Write 9 New Done Done false 6685 Acked 7684 Read 8 New Done Done false 6686 Acked 7685 Read 4 New Done Done false 6687 Acked 7686 Flush 0 New Done Done false 6688 Acked 7687 Read 7 New Done Done false 6689 Acked 7688 Flush 0 New Done Done false 6690 Acked 7689 Read 1 New Done Done false 6691 Acked 7690 Write 6 New Done Done false 6692 Acked 7691 Flush 0 New Done Done false 6693 Acked 7692 Flush 0 New Done Done false 6694 Acked 7693 Read 1 New Done Done false 6695 Acked 7694 Write 2 New Done Done false 6696 Acked 7695 Read 9 New Done Done false 6697 Acked 7696 Read 6 New Done Done false 6698 Acked 7697 Read 6 New Done Done false 6699 Acked 7698 Write 9 New Done Done false 6700 Acked 7699 Write 6 New Done Done false 6701 Acked 7700 Read 5 New Done Done false 6702 Acked 7701 Read 7 New Done Done false 6703 Acked 7702 Read 8 New Done Done false 6704 Acked 7703 Write 4 New Done Done false 6705 Acked 7704 Write 8 New Done Done false 6706 Acked 7705 Flush 0 New Done Done false 6707 Acked 7706 Flush 0 New Done Done false 6708 Acked 7707 Write 6 New Done Done false 6709 Acked 7708 Read 1 New Done Done false 6710 Acked 7709 Write 10 New Done Done false 6711 Acked 7710 Flush 0 New Done Done false 6712 Acked 7711 Read 4 New Done Done false 6713 Acked 7712 Read 2 New Done Done false 6714 Acked 7713 Write 9 New Done Done false 6715 Acked 7714 Flush 0 New Done Done false 6716 Acked 7715 Write 6 New Done Done false 6717 Acked 7716 Read 9 New Done Done false 6718 Acked 7717 Write 5 New Done Done false 6719 Acked 7718 Read 9 New Done Done false 6720 Acked 7719 Write 5 New Done Done false 6721 Acked 7720 Write 8 New Done Done false 6722 Acked 7721 Write 6 New Done Done false 6723 Acked 7722 Read 6 New Done Done false 6724 Acked 7723 Read 3 New Done Done false 6725 Acked 7724 Write 10 New Done Done false 6726 Acked 7725 Read 10 New Done Done false 6727 Acked 7726 Read 5 New Done Done false 6728 Acked 7727 Flush 0 New Done Done false 6729 Acked 7728 Read 3 New Done Done false 6730 Acked 7729 Read 3 New Done Done false 6731 Acked 7730 Flush 0 New Done Done false 6732 Acked 7731 Read 8 New Done Done false 6733 Acked 7732 Write 2 New Done Done false 6734 Acked 7733 Read 9 New Done Done false 6735 Acked 7734 Write 6 New Done Done false 6736 Acked 7735 Write 7 New Done Done false 6737 Acked 7736 Write 7 New Done Done false 6738 Acked 7737 Write 3 New Done Done false 6739 Acked 7738 Write 1 New Done Done false 6740 Acked 7739 Read 10 New Done Done false 6741 Acked 7740 Write 5 New Done Done false 6742 Acked 7741 Flush 0 New Done Done false 6743 Acked 7742 Flush 0 New Done Done false 6744 Acked 7743 Write 5 New Done Done false 6745 Acked 7744 Write 3 New Done Done false 6746 Acked 7745 Read 10 New Done Done false 6747 Acked 7746 Read 8 New Done Done false 6748 Acked 7747 Read 5 New Done Done false 6749 Acked 7748 Read 5 New Done Done false 6750 Acked 7749 Write 1 New Done Done false 6751 Acked 7750 Flush 0 New Done Done false 6752 Acked 7751 Read 3 New Done Done false 6753 Acked 7752 Read 7 New Done Done false 6754 Acked 7753 Write 3 New Done Done false 6755 Acked 7754 Flush 0 New Done Done false 6756 Acked 7755 Write 2 New Done Done false 6757 Acked 7756 Write 4 New Done Done false 6758 Acked 7757 Write 5 New Done Done false 6759 Acked 7758 Read 1 New Done Done false 6760 Acked 7759 Read 4 New Done Done false 6761 Acked 7760 Write 1 New Done Done false 6762 Acked 7761 Write 9 New Done Done false 6763 Acked 7762 Read 5 New Done Done false 6764 Acked 7763 Read 2 New Done Done false 6765 Acked 7764 Write 2 New Done Done false 6766 Acked 7765 Flush 0 New Done Done false 6767 Acked 7766 Read 2 New Done Done false 6768 Acked 7767 Write 10 New Done Done false 6769 Acked 7768 Flush 0 New Done Done false 6770 Acked 7769 Read 9 New Done Done false 6771 Acked 7770 Read 8 New Done Done false 6772 Acked 7771 Read 5 New Done Done false 6773 Acked 7772 Read 3 New Done Done false 6774 Acked 7773 Read 6 New Done Done false 6775 Acked 7774 Read 7 New Done Done false 6776 Acked 7775 Write 1 New Done Done false 6777 Acked 7776 Write 7 New Done Done false 6778 Acked 7777 Write 6 New Done Done false 6779 Acked 7778 Write 6 New Done Done false 6780 Acked 7779 Write 7 New Done Done false 6781 Acked 7780 Read 2 New Done Done false 6782 Acked 7781 Read 6 New Done Done false 6783 Acked 7782 Read 2 New Done Done false 6784 Acked 7783 Write 5 New Done Done false 6785 Acked 7784 Read 4 New Done Done false 6786 Acked 7785 Write 1 New Done Done false 6787 Acked 7786 Write 3 New Done Done false 6788 Acked 7787 Write 1 New Done Done false 6789 Acked 7788 Write 4 New Done Done false 6790 Acked 7789 Read 4 New Done Done false 6791 Acked 7790 Write 6 New Done Done false 6792 Acked 7791 Write 3 New Done Done false 6793 Acked 7792 Write 4 New Done Done false 6794 Acked 7793 Read 5 New Done Done false 6795 Acked 7794 Read 3 New Done Done false 6796 Acked 7795 Read 4 New Done Done false 6797 Acked 7796 Write 9 New Done Done false 6798 Acked 7797 Write 1 New Done Done false 6799 Acked 7798 Write 8 New Done Done false 6800 Acked 7799 Read 9 New Done Done false 6801 Acked 7800 Write 5 New Done Done false 6802 Acked 7801 Write 5 New Done Done false 6803 Acked 7802 Write 3 New Done Done false 6804 Acked 7803 Read 10 New Done Done false 6805 Acked 7804 Read 1 New Done Done false 6806 Acked 7805 Read 6 New Done Done false 6807 Acked 7806 Write 9 New Done Done false 6808 Acked 7807 Read 10 New Done Done false 6809 Acked 7808 Write 1 New Done Done false 6810 Acked 7809 Read 3 New Done Done false 6811 Acked 7810 Read 6 New Done Done false 6812 Acked 7811 Write 6 New Done Done false 6813 Acked 7812 Read 10 New Done Done false 6814 Acked 7813 Read 5 New Done Done false 6815 Acked 7814 Read 3 New Done Done false 6816 Acked 7815 Write 10 New Done Done false 6817 Acked 7816 Read 9 New Done Done false 6818 Acked 7817 Flush 0 New Done Done false 6819 Acked 7818 Read 4 New Done Done false 6820 Acked 7819 Read 8 New Done Done false 6821 Acked 7820 Read 9 New Done Done false 6822 Acked 7821 Read 8 New Done Done false 6823 Acked 7822 Write 3 New Done Done false 6824 Acked 7823 Read 10 New Done Done false 6825 Acked 7824 Flush 0 New Done Done false 6826 Acked 7825 Read 4 New Done Done false 6827 Acked 7826 Read 6 New Done Done false 6828 Acked 7827 Read 3 New Done Done false 6829 Acked 7828 Read 6 New Done Done false 6830 Acked 7829 Read 5 New Done Done false 6831 Acked 7830 Write 9 New Done Done false 6832 Acked 7831 Write 2 New Done Done false 6833 Acked 7832 Write 3 New Done Done false 6834 Acked 7833 Read 10 New Done Done false 6835 Acked 7834 Flush 0 New Done Done false 6836 Acked 7835 Write 1 New Done Done false 6837 Acked 7836 Read 1 New Done Done false 6838 Acked 7837 Read 6 New Done Done false 6839 Acked 7838 Read 7 New Done Done false 6840 Acked 7839 Write 2 New Done Done false 6841 Acked 7840 Flush 0 New Done Done false 6842 Acked 7841 Write 4 New Done Done false 6843 Acked 7842 Read 8 New Done Done false 6844 Acked 7843 Write 2 New Done Done false 6845 Acked 7844 Write 3 New Done Done false 6846 Acked 7845 Write 5 New Done Done false 6847 Acked 7846 Write 8 New Done Done false 6848 Acked 7847 Write 5 New Done Done false 6849 Acked 7848 Write 3 New Done Done false 6850 Acked 7849 Write 7 New Done Done false 6851 Acked 7850 Write 7 New Done Done false 6852 Acked 7851 Read 8 New Done Done false 6853 Acked 7852 Read 9 New Done Done false 6854 Acked 7853 Flush 0 New Done Done false 6855 Acked 7854 Flush 0 New Done Done false 6856 Acked 7855 Read 4 New Done Done false 6857 Acked 7856 Write 10 New Done Done false 6858 Acked 7857 Read 2 New Done Done false 6859 Acked 7858 Write 2 New Done Done false 6860 Acked 7859 Write 8 New Done Done false 6861 Acked 7860 Flush 0 New Done Done false 6862 Acked 7861 Read 10 New Done Done false 6863 Acked 7862 Write 10 New Done Done false 6864 Acked 7863 Read 2 New Done Done false 6865 Acked 7864 Flush 0 New Done Done false 6866 Acked 7865 Read 10 New Done Done false 6867 Acked 7866 Read 4 New Done Done false 6868 Acked 7867 Write 6 New Done Done false 6869 Acked 7868 Write 7 New Done Done false 6870 Acked 7869 Flush 0 New Done Done false 6871 Acked 7870 Write 5 New Done Done false 6872 Acked 7871 Read 9 New Done Done false 6873 Acked 7872 Write 9 New Done Done false 6874 Acked 7873 Read 4 New Done Done false 6875 Acked 7874 Write 1 New Done Done false 6876 Acked 7875 Flush 0 New Done Done false 6877 Acked 7876 Read 7 New Done Done false 6878 Acked 7877 Write 9 New Done Done false 6879 Acked 7878 Read 6 New Done Done false 6880 Acked 7879 Write 3 New Done Done false 6881 Acked 7880 Read 8 New Done Done false 6882 Acked 7881 Read 7 New Done Done false 6883 Acked 7882 Write 2 New Done Done false 6884 Acked 7883 Read 4 New Done Done false 6885 Acked 7884 Write 6 New Done Done false 6886 Acked 7885 Write 9 New Done Done false 6887 Acked 7886 Write 8 New Done Done false 6888 Acked 7887 Read 9 New Done Done false 6889 Acked 7888 Write 8 New Done Done false 6890 Acked 7889 Write 8 New Done Done false 6891 Acked 7890 Write 10 New Done Done false 6892 Acked 7891 Write 9 New Done Done false 6893 Acked 7892 Read 1 New Done Done false 6894 Acked 7893 Read 9 New Done Done false 6895 Acked 7894 Read 8 New Done Done false 6896 Acked 7895 Write 10 New Done Done false 6897 Acked 7896 Read 9 New Done Done false 6898 Acked 7897 Read 3 New Done Done false 6899 Acked 7898 Read 1 New Done Done false 6900 Acked 7899 Read 2 New Done Done false 6901 Acked 7900 Read 6 New Done Done false 6902 Acked 7901 Flush 0 New Done Done false 6903 Acked 7902 Write 2 New Done Done false 6904 Acked 7903 Flush 0 New Done Done false 6905 Acked 7904 Flush 0 New Done Done false 6906 Acked 7905 Write 1 New Done Done false 6907 Acked 7906 Read 6 New Done Done false 6908 Acked 7907 Write 2 New Done Done false 6909 Acked 7908 Write 1 New Done Done false 6910 Acked 7909 Read 5 New Done Done false 6911 Acked 7910 Read 3 New Done Done false 6912 Acked 7911 Write 4 New Done Done false 6913 Acked 7912 Write 7 New Done Done false 6914 Acked 7913 Write 5 New Done Done false 6915 Acked 7914 Read 7 New Done Done false 6916 Acked 7915 Read 7 New Done Done false 6917 Acked 7916 Read 2 New Done Done false 6918 Acked 7917 Read 9 New Done Done false 6919 Acked 7918 Read 2 New Done Done false 6920 Acked 7919 Write 7 New Done Done false 6921 Acked 7920 Read 2 New Done Done false 6922 Acked 7921 Read 7 New Done Done false 6923 Acked 7922 Read 10 New Done Done false 6924 Acked 7923 Write 7 New Done Done false 6925 Acked 7924 Flush 0 New Done Done false 6926 Acked 7925 Read 2 New Done Done false 6927 Acked 7926 Read 6 New Done Done false 6928 Acked 7927 Flush 0 New Done Done false 6929 Acked 7928 Read 8 New Done Done false 6930 Acked 7929 Write 8 New Done Done false 6931 Acked 7930 Flush 0 New Done Done false 6932 Acked 7931 Write 3 New Done Done false 6933 Acked 7932 Write 8 New Done Done false 6934 Acked 7933 Write 6 New Done Done false 6935 Acked 7934 Write 3 New Done Done false 6936 Acked 7935 Read 2 New Done Done false 6937 Acked 7936 Read 5 New Done Done false 6938 Acked 7937 Read 3 New Done Done false 6939 Acked 7938 Write 7 New Done Done false 6940 Acked 7939 Read 8 New Done Done false 6941 Acked 7940 Read 9 New Done Done false 6942 Acked 7941 Write 8 New Done Done false 6943 Acked 7942 Write 7 New Done Done false 6944 Acked 7943 Read 1 New Done Done false 6945 NotAcked 7944 Flush 0 New Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 1 1 2 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7642 7930 7930 Downstairs last five completed: 7642 7641 7640 7639 7638 Upstairs last five completed: 6944 6943 6942 6941 6940 CLIENT: Up:1 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:04.300041507Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:04.300111462Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:04.300139921Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:04.300166702Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:04.300187283Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:04.300230519Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:04.300484478Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:05.301493199Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:05.301565401Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:05.30159405Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:05.30162125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:05.301651885Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:05.301693656Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:05.301969437Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:06.303078873Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:06.30314316Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:06.303170568Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:06.303204204Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:06.303225557Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:06.303324207Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:06.303574914Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:07.304345535Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:07.304454332Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:07.304529453Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:07.304553944Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:07.304590237Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:07.304611532Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:07.304634354Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:07.304868047Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 6644 Acked 7643 Write 1 New Done Done true 6645 Acked 7644 Read 4 New Done Done true 6646 Acked 7645 Write 1 New Done Done true 6647 Acked 7646 Read 1 New Done Done true 6648 Acked 7647 Write 9 New Done Done true 6649 Acked 7648 Write 4 New Done Done true 6650 Acked 7649 Read 10 New Done Done true 6651 Acked 7650 Read 3 New Done Done true 6652 Acked 7651 Write 5 New Done Done true 6653 Acked 7652 Flush 0 New Done Done true 6654 Acked 7653 Flush 0 New Done Done true 6655 Acked 7654 Flush 0 New Done Done true 6656 Acked 7655 Read 1 New Done Done true 6657 Acked 7656 Read 8 New Done Done true 6658 Acked 7657 Read 1 New Done Done true 6659 Acked 7658 Flush 0 New Done Done true 6660 Acked 7659 Read 3 New Done Done true 6661 Acked 7660 Read 5 New Done Done true 6662 Acked 7661 Flush 0 New Done Done true 6663 Acked 7662 Read 1 New Done Done true 6664 Acked 7663 Write 7 New Done Done true 6665 Acked 7664 Write 5 New Done Done true 6666 Acked 7665 Read 9 New Done Done true 6667 Acked 7666 Write 8 New Done Done true 6668 Acked 7667 Read 8 New Done Done true 6669 Acked 7668 Read 5 New Done Done true 6670 Acked 7669 Read 5 New Done Done true 6671 Acked 7670 Write 2 New Done Done true 6672 Acked 7671 Read 6 New Done Done true 6673 Acked 7672 Write 7 New Done Done true 6674 Acked 7673 Read 2 New Done Done true 6675 Acked 7674 Read 6 New Done Done true 6676 Acked 7675 Read 3 New Done Done true 6677 Acked 7676 Read 3 New Done Done true 6678 Acked 7677 Read 1 New Done Done true 6679 Acked 7678 Read 2 New Done Done true 6680 Acked 7679 Flush 0 New Done Done true 6681 Acked 7680 Write 7 New Done Done true 6682 Acked 7681 Write 5 New Done Done true 6683 Acked 7682 Read 4 New Done Done true 6684 Acked 7683 Write 9 New Done Done true 6685 Acked 7684 Read 8 New Done Done true 6686 Acked 7685 Read 4 New Done Done true 6687 Acked 7686 Flush 0 New Done Done true 6688 Acked 7687 Read 7 New Done Done true 6689 Acked 7688 Flush 0 New Done Done true 6690 Acked 7689 Read 1 New Done Done true 6691 Acked 7690 Write 6 New Done Done true 6692 Acked 7691 Flush 0 New Done Done true 6693 Acked 7692 Flush 0 New Done Done true 6694 Acked 7693 Read 1 New Done Done true 6695 Acked 7694 Write 2 New Done Done true 6696 Acked 7695 Read 9 New Done Done true 6697 Acked 7696 Read 6 New Done Done true 6698 Acked 7697 Read 6 New Done Done true 6699 Acked 7698 Write 9 New Done Done true 6700 Acked 7699 Write 6 New Done Done true 6701 Acked 7700 Read 5 New Done Done true 6702 Acked 7701 Read 7 New Done Done true 6703 Acked 7702 Read 8 New Done Done true 6704 Acked 7703 Write 4 New Done Done true 6705 Acked 7704 Write 8 New Done Done true 6706 Acked 7705 Flush 0 New Done Done true 6707 Acked 7706 Flush 0 New Done Done true 6708 Acked 7707 Write 6 New Done Done true 6709 Acked 7708 Read 1 New Done Done true 6710 Acked 7709 Write 10 New Done Done true 6711 Acked 7710 Flush 0 New Done Done true 6712 Acked 7711 Read 4 New Done Done true 6713 Acked 7712 Read 2 New Done Done true 6714 Acked 7713 Write 9 New Done Done true 6715 Acked 7714 Flush 0 New Done Done true 6716 Acked 7715 Write 6 New Done Done true 6717 Acked 7716 Read 9 New Done Done true 6718 Acked 7717 Write 5 New Done Done true 6719 Acked 7718 Read 9 New Done Done true 6720 Acked 7719 Write 5 New Done Done true 6721 Acked 7720 Write 8 New Done Done true 6722 Acked 7721 Write 6 New Done Done true 6723 Acked 7722 Read 6 New Done Done true 6724 Acked 7723 Read 3 New Done Done true 6725 Acked 7724 Write 10 New Done Done true 6726 Acked 7725 Read 10 New Done Done true 6727 Acked 7726 Read 5 New Done Done true 6728 Acked 7727 Flush 0 New Done Done true 6729 Acked 7728 Read 3 New Done Done true 6730 Acked 7729 Read 3 New Done Done true 6731 Acked 7730 Flush 0 New Done Done true 6732 Acked 7731 Read 8 New Done Done true 6733 Acked 7732 Write 2 New Done Done true 6734 Acked 7733 Read 9 New Done Done true 6735 Acked 7734 Write 6 New Done Done true 6736 Acked 7735 Write 7 New Done Done true 6737 Acked 7736 Write 7 New Done Done true 6738 Acked 7737 Write 3 New Done Done true 6739 Acked 7738 Write 1 New Done Done true 6740 Acked 7739 Read 10 New Done Done true 6741 Acked 7740 Write 5 New Done Done true 6742 Acked 7741 Flush 0 New Done Done true 6743 Acked 7742 Flush 0 New Done Done true 6744 Acked 7743 Write 5 New Done Done true 6745 Acked 7744 Write 3 New Done Done true 6746 Acked 7745 Read 10 New Done Done true 6747 Acked 7746 Read 8 New Done Done true 6748 Acked 7747 Read 5 New Done Done true 6749 Acked 7748 Read 5 New Done Done true 6750 Acked 7749 Write 1 New Done Done true 6751 Acked 7750 Flush 0 New Done Done true 6752 Acked 7751 Read 3 New Done Done true 6753 Acked 7752 Read 7 New Done Done true 6754 Acked 7753 Write 3 New Done Done true 6755 Acked 7754 Flush 0 New Done Done true 6756 Acked 7755 Write 2 New Done Done true 6757 Acked 7756 Write 4 New Done Done true 6758 Acked 7757 Write 5 New Done Done true 6759 Acked 7758 Read 1 New Done Done true 6760 Acked 7759 Read 4 New Done Done true 6761 Acked 7760 Write 1 New Done Done true 6762 Acked 7761 Write 9 New Done Done true 6763 Acked 7762 Read 5 New Done Done true 6764 Acked 7763 Read 2 New Done Done true 6765 Acked 7764 Write 2 New Done Done true 6766 Acked 7765 Flush 0 New Done Done true 6767 Acked 7766 Read 2 New Done Done true 6768 Acked 7767 Write 10 New Done Done true 6769 Acked 7768 Flush 0 New Done Done true 6770 Acked 7769 Read 9 New Done Done true 6771 Acked 7770 Read 8 New Done Done true 6772 Acked 7771 Read 5 New Done Done true 6773 Acked 7772 Read 3 New Done Done true 6774 Acked 7773 Read 6 New Done Done true 6775 Acked 7774 Read 7 New Done Done true 6776 Acked 7775 Write 1 New Done Done true 6777 Acked 7776 Write 7 New Done Done true 6778 Acked 7777 Write 6 New Done Done true 6779 Acked 7778 Write 6 New Done Done true 6780 Acked 7779 Write 7 New Done Done true 6781 Acked 7780 Read 2 New Done Done true 6782 Acked 7781 Read 6 New Done Done true 6783 Acked 7782 Read 2 New Done Done true 6784 Acked 7783 Write 5 New Done Done true 6785 Acked 7784 Read 4 New Done Done true 6786 Acked 7785 Write 1 New Done Done true 6787 Acked 7786 Write 3 New Done Done true 6788 Acked 7787 Write 1 New Done Done true 6789 Acked 7788 Write 4 New Done Done true 6790 Acked 7789 Read 4 New Done Done true 6791 Acked 7790 Write 6 New Done Done true 6792 Acked 7791 Write 3 New Done Done true 6793 Acked 7792 Write 4 New Done Done true 6794 Acked 7793 Read 5 New Done Done true 6795 Acked 7794 Read 3 New Done Done true 6796 Acked 7795 Read 4 New Done Done true 6797 Acked 7796 Write 9 New Done Done true 6798 Acked 7797 Write 1 New Done Done true 6799 Acked 7798 Write 8 New Done Done true 6800 Acked 7799 Read 9 New Done Done true 6801 Acked 7800 Write 5 New Done Done true 6802 Acked 7801 Write 5 New Done Done true 6803 Acked 7802 Write 3 New Done Done true 6804 Acked 7803 Read 10 New Done Done true 6805 Acked 7804 Read 1 New Done Done true 6806 Acked 7805 Read 6 New Done Done true 6807 Acked 7806 Write 9 New Done Done true 6808 Acked 7807 Read 10 New Done Done true 6809 Acked 7808 Write 1 New Done Done true 6810 Acked 7809 Read 3 New Done Done true 6811 Acked 7810 Read 6 New Done Done true 6812 Acked 7811 Write 6 New Done Done true 6813 Acked 7812 Read 10 New Done Done true 6814 Acked 7813 Read 5 New Done Done true 6815 Acked 7814 Read 3 New Done Done true 6816 Acked 7815 Write 10 New Done Done true 6817 Acked 7816 Read 9 New Done Done true 6818 Acked 7817 Flush 0 New Done Done true 6819 Acked 7818 Read 4 New Done Done true 6820 Acked 7819 Read 8 New Done Done true 6821 Acked 7820 Read 9 New Done Done true 6822 Acked 7821 Read 8 New Done Done true 6823 Acked 7822 Write 3 New Done Done true 6824 Acked 7823 Read 10 New Done Done true 6825 Acked 7824 Flush 0 New Done Done true 6826 Acked 7825 Read 4 New Done Done true 6827 Acked 7826 Read 6 New Done Done true 6828 Acked 7827 Read 3 New Done Done true 6829 Acked 7828 Read 6 New Done Done true 6830 Acked 7829 Read 5 New Done Done true 6831 Acked 7830 Write 9 New Done Done true 6832 Acked 7831 Write 2 New Done Done true 6833 Acked 7832 Write 3 New Done Done true 6834 Acked 7833 Read 10 New Done Done true 6835 Acked 7834 Flush 0 New Done Done true 6836 Acked 7835 Write 1 New Done Done true 6837 Acked 7836 Read 1 New Done Done true 6838 Acked 7837 Read 6 New Done Done true 6839 Acked 7838 Read 7 New Done Done true 6840 Acked 7839 Write 2 New Done Done true 6841 Acked 7840 Flush 0 New Done Done true 6842 Acked 7841 Write 4 New Done Done true 6843 Acked 7842 Read 8 New Done Done true 6844 Acked 7843 Write 2 New Done Done true 6845 Acked 7844 Write 3 New Done Done true 6846 Acked 7845 Write 5 New Done Done true 6847 Acked 7846 Write 8 New Done Done true 6848 Acked 7847 Write 5 New Done Done true 6849 Acked 7848 Write 3 New Done Done true 6850 Acked 7849 Write 7 New Done Done true 6851 Acked 7850 Write 7 New Done Done true 6852 Acked 7851 Read 8 New Done Done true 6853 Acked 7852 Read 9 New Done Done true 6854 Acked 7853 Flush 0 New Done Done true 6855 Acked 7854 Flush 0 New Done Done true 6856 Acked 7855 Read 4 New Done Done true 6857 Acked 7856 Write 10 New Done Done true 6858 Acked 7857 Read 2 New Done Done true 6859 Acked 7858 Write 2 New Done Done true 6860 Acked 7859 Write 8 New Done Done true 6861 Acked 7860 Flush 0 New Done Done true 6862 Acked 7861 Read 10 New Done Done true 6863 Acked 7862 Write 10 New Done Done true 6864 Acked 7863 Read 2 New Done Done true 6865 Acked 7864 Flush 0 New Done Done true 6866 Acked 7865 Read 10 New Done Done true 6867 Acked 7866 Read 4 New Done Done true 6868 Acked 7867 Write 6 New Done Done true 6869 Acked 7868 Write 7 New Done Done true 6870 Acked 7869 Flush 0 New Done Done true 6871 Acked 7870 Write 5 New Done Done true 6872 Acked 7871 Read 9 New Done Done true 6873 Acked 7872 Write 9 New Done Done true 6874 Acked 7873 Read 4 New Done Done true 6875 Acked 7874 Write 1 New Done Done true 6876 Acked 7875 Flush 0 New Done Done true 6877 Acked 7876 Read 7 New Done Done true 6878 Acked 7877 Write 9 New Done Done true 6879 Acked 7878 Read 6 New Done Done true 6880 Acked 7879 Write 3 New Done Done true 6881 Acked 7880 Read 8 New Done Done true 6882 Acked 7881 Read 7 New Done Done true 6883 Acked 7882 Write 2 New Done Done true 6884 Acked 7883 Read 4 New Done Done true 6885 Acked 7884 Write 6 New Done Done true 6886 Acked 7885 Write 9 New Done Done true 6887 Acked 7886 Write 8 New Done Done true 6888 Acked 7887 Read 9 New Done Done true 6889 Acked 7888 Write 8 New Done Done true 6890 Acked 7889 Write 8 New Done Done true 6891 Acked 7890 Write 10 New Done Done true 6892 Acked 7891 Write 9 New Done Done true 6893 Acked 7892 Read 1 New Done Done true 6894 Acked 7893 Read 9 New Done Done true 6895 Acked 7894 Read 8 New Done Done true 6896 Acked 7895 Write 10 New Done Done true 6897 Acked 7896 Read 9 New Done Done true 6898 Acked 7897 Read 3 New Done Done true 6899 Acked 7898 Read 1 New Done Done true 6900 Acked 7899 Read 2 New Done Done true 6901 Acked 7900 Read 6 New Done Done true 6902 Acked 7901 Flush 0 New Done Done true 6903 Acked 7902 Write 2 New Done Done true 6904 Acked 7903 Flush 0 New Done Done true 6905 Acked 7904 Flush 0 New Done Done true 6906 Acked 7905 Write 1 New Done Done true 6907 Acked 7906 Read 6 New Done Done true 6908 Acked 7907 Write 2 New Done Done true 6909 Acked 7908 Write 1 New Done Done true 6910 Acked 7909 Read 5 New Done Done true 6911 Acked 7910 Read 3 New Done Done true 6912 Acked 7911 Write 4 New Done Done true 6913 Acked 7912 Write 7 New Done Done true 6914 Acked 7913 Write 5 New Done Done true 6915 Acked 7914 Read 7 New Done Done true 6916 Acked 7915 Read 7 New Done Done true 6917 Acked 7916 Read 2 New Done Done true 6918 Acked 7917 Read 9 New Done Done true 6919 Acked 7918 Read 2 New Done Done true 6920 Acked 7919 Write 7 New Done Done true 6921 Acked 7920 Read 2 New Done Done true 6922 Acked 7921 Read 7 New Done Done true 6923 Acked 7922 Read 10 New Done Done true 6924 Acked 7923 Write 7 New Done Done true 6925 Acked 7924 Flush 0 New Done Done true 6926 Acked 7925 Read 2 New Done Done true 6927 Acked 7926 Read 6 New Done Done true 6928 Acked 7927 Flush 0 New Done Done true 6929 Acked 7928 Read 8 New Done Done true 6930 Acked 7929 Write 8 New Done Done true 6931 Acked 7930 Flush 0 New Done Done true 6932 Acked 7931 Write 3 New Done Done true 6933 Acked 7932 Write 8 New Done Done true 6934 Acked 7933 Write 6 New Done Done true 6935 Acked 7934 Write 3 New Done Done true 6936 Acked 7935 Read 2 New Done Done true 6937 Acked 7936 Read 5 New Done Done true 6938 Acked 7937 Read 3 New Done Done true 6939 Acked 7938 Write 7 New Done Done true 6940 Acked 7939 Read 8 New Done Done true 6941 Acked 7940 Read 9 New Done Done true 6942 Acked 7941 Write 8 New Done Done true 6943 Acked 7942 Write 7 New Done Done true 6944 Acked 7943 Read 1 New Done Done true 6945 Acked 7944 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 7642 7944 7944 Downstairs last five completed: 7642 7641 7640 7639 7638 Upstairs last five completed: 6945 6944 6943 6942 6941 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:08.30596551Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:08.306032461Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:08.306060308Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:08.306087035Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:08.306107288Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:08.306190179Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:08.306439857Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:09.307486873Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:09.30755358Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:09.30758243Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:09.307613514Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:09.307634739Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1208) {"msg":"[0] client re-new 302 jobs since flush 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:09.307748193Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810][0] reports Running {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:09.308140885Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.308237729Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.308396083Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.31040245Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.310445155Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.310470233Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 7642","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.310795146Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.310825862Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.310849738Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.310872446Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:10.310903287Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 7944 7943 7942 7941 7940 Upstairs last five completed: 6945 6944 6943 6942 6941 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(2) stop 2 [8830] Got stop action so:false kr:false Jan 02 22:35:11.880 INFO request completed, latency_us: 478, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5d912eed-f7ce-47d3-bb79-65ac07876944, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:11.881527871Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:35:11.881 INFO request completed, latency_us: 170, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5de450cd-1685-4b1e-81de-2d392a106056, remote_addr: 127.0.0.1:37434, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:11.881667224Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} local_addr: 127.0.0.1:9998 {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:11.881728878Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:11.881771406Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:11.881817615Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:11.88186019Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:12.882161087Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:12.882232545Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:12.882261035Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:12.882287843Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:12.882308338Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:12.882355717Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:12.882456529Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:13.883604053Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:13.883734624Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:13.883763961Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:13.883790706Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:13.883811182Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:13.883875445Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:13.883922022Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:14.885160151Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:14.885231587Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:14.885259561Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:14.885286287Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:14.885306772Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:14.885375554Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:14.885420557Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:35:15.884 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/2, method: GET, req_id: f221464c-58d4-460c-a9fc-995d4a0a4fb2, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 3281 len 32768 data: 2 2 3 3 3 3 2 3 002/300 Read block 3374 len 4096 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:15.88691622Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:15.887015318Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:15.8870721Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:15.887124201Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:15.887150334Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 2 jobs since flush 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:15.887173041Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:15.88719512Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Write block 1760 len 8192 data: 8 6 004/300 Read block 1528 len 28672 005/300 Write block 2116 len 8192 data: 4 5 006/300 Read block 4076 len 32768 007/300 Flush 008/300 Write block 2574 len 40960 data: 7 7 7 8 8 8 7 6 6 5 009/300 Write block 2733 len 4096 data: 7 010/300 Write block 17 len 16384 data: 7 6 5 5 011/300 Write block 936 len 4096 data: 6 012/300 Read block 4689 len 28672 013/300 Write block 4746 len 20480 data: 4 5 5 6 6 014/300 Flush 015/300 Read block 474 len 36864 016/300 Read block 2593 len 28672 017/300 Read block 535 len 4096 018/300 Read block 4648 len 4096 019/300 Flush 020/300 Write block 366 len 16384 data: 9 10 7 5 021/300 Read block 4556 len 4096 022/300 Read block 1949 len 20480 023/300 Write block 428 len 16384 data: 7 8 7 8 024/300 Write block 2467 len 12288 data: 6 7 6 025/300 Flush 026/300 Write block 2195 len 32768 data: 6 8 7 7 7 7 7 5 027/300 Write block 2903 len 8192 data: 6 5 028/300 Read block 3775 len 40960 029/300 Read block 962 len 20480 030/300 Read block 4424 len 4096 031/300 Read block 811 len 4096 032/300 Read block 3338 len 28672 033/300 Write block 1345 len 20480 data: 6 7 6 7 8 034/300 Read block 1009 len 24576 035/300 Write block 4027 len 28672 data: 4 2 2 2 2 2 3 036/300 Read block 4912 len 16384 037/300 Write block 273 len 8192 data: 8 7 038/300 Read block 2059 len 4096 039/300 Write block 1013 len 28672 data: 5 6 4 4 4 4 4 040/300 Flush 041/300 Read block 638 len 20480 042/300 Read block 3406 len 28672 043/300 Write block 672 len 28672 data: 4 3 4 5 5 4 5 044/300 Read block 4239 len 36864 045/300 Read block 166 len 8192 046/300 Read block 3927 len 4096 047/300 Read block 1610 len 36864 048/300 Read block 1335 len 16384 049/300 Write block 4043 len 24576 data: 7 6 7 7 8 7 050/300 Write block 4363 len 20480 data: 4 4 4 3 3 051/300 Write block 1803 len 4096 data: 4 052/300 Read block 86 len 40960 053/300 Write block 726 len 8192 data: 7 7 054/300 Read block 4066 len 36864 055/300 Write block 1687 len 28672 data: 9 10 7 7 7 6 5 056/300 Read block 1596 len 40960 057/300 Read block 4762 len 32768 058/300 Write block 1179 len 28672 data: 5 5 6 5 4 4 3 059/300 Write block 1253 len 40960 data: 7 6 7 8 8 8 7 7 8 6 060/300 Write block 1346 len 20480 data: 8 7 8 9 9 061/300 Read block 656 len 4096 062/300 Write block 4606 len 20480 data: 7 6 7 8 8 063/300 Read block 3297 len 36864 064/300 Read block 3818 len 12288 065/300 Write block 2178 len 8192 data: 4 4 066/300 Read block 4483 len 40960 067/300 Read block 2045 len 20480 068/300 Read block 2864 len 28672 069/300 Read block 441 len 12288 070/300 Write block 4509 len 4096 data: 6 071/300 Write block 1340 len 28672 data: 4 5 5 5 6 7 9 072/300 Read block 4553 len 8192 073/300 Read block 3931 len 8192 074/300 Read block 3904 len 32768 075/300 Read block 4150 len 20480 076/300 Read block 183 len 36864 077/300 Write block 2017 len 28672 data: 5 4 3 3 3 3 3 078/300 Write block 4193 len 16384 data: 4 5 4 3 079/300 Read block 924 len 36864 080/300 Write block 1602 len 16384 data: 3 3 3 4 081/300 Read block 1176 len 20480 082/300 Write block 2184 len 24576 data: 2 3 3 3 4 5 083/300 Write block 967 len 20480 data: 5 5 5 5 5 084/300 Read block 1126 len 20480 085/300 Read block 3483 len 36864 086/300 Read block 914 len 28672 087/300 Flush 088/300 Read block 2869 len 16384 089/300 Read block 4026 len 40960 090/300 Flush 091/300 Read block 1931 len 20480 092/300 Write block 4983 len 40960 data: 7 6 5 6 5 4 4 3 3 4 093/300 Read block 284 len 24576 094/300 Read block 1084 len 28672 095/300 Read block 626 len 20480 096/300 Write block 4384 len 36864 data: 5 6 6 4 4 4 3 3 2 097/300 Read block 1159 len 16384 098/300 Write block 3493 len 24576 data: 2 2 2 3 4 3 099/300 Write block 1261 len 40960 data: 9 7 7 6 7 6 5 4 4 5 100/300 Flush 101/300 Flush 102/300 Write block 1250 len 40960 data: 4 3 3 8 7 8 9 9 9 8 103/300 Read block 1101 len 36864 104/300 Read block 3642 len 12288 105/300 Write block 4288 len 8192 data: 4 4 106/300 Read block 2177 len 32768 107/300 Write block 545 len 36864 data: 5 6 7 7 6 6 5 3 3 108/300 Read block 3467 len 20480 109/300 Write block 3465 len 16384 data: 3 3 3 4 110/300 Read block 693 len 28672 111/300 Flush 112/300 Flush 113/300 Flush 114/300 Flush 115/300 Flush 116/300 Read block 2835 len 16384 117/300 Flush 118/300 Read block 487 len 20480 119/300 Read block 4158 len 12288 120/300 Read block 716 len 4096 121/300 Read block 3377 len 4096 122/300 Write block 1391 len 12288 data: 7 10 8 123/300 Write block 2810 len 36864 data: 3 2 2 2 2 3 6 4 6 124/300 Read block 4480 len 28672 125/300 Read block 3601 len 16384 126/300 Write block 3215 len 20480 data: 7 5 5 5 4 127/300 Write block 3363 len 12288 data: 6 5 5 128/300 Read block 562 len 36864 129/300 Read block 3710 len 20480 130/300 Read block 906 len 16384 131/300 Write block 2221 len 4096 data: 3 132/300 Write block 3342 len 8192 data: 4 5 133/300 Read block 3620 len 28672 134/300 Write block 2579 len 8192 data: 9 8 135/300 Flush 136/300 Read block 4464 len 12288 137/300 Flush 138/300 Write block 283 len 28672 data: 8 7 5 7 7 7 7 139/300 Write block 3166 len 4096 data: 3 140/300 Write block 2933 len 16384 data: 5 7 6 7 141/300 Flush 142/300 Read block 1990 len 36864 143/300 Write block 2416 len 20480 data: 9 9 9 8 8 144/300 Write block 3572 len 36864 data: 3 3 3 4 4 4 4 3 3 145/300 Flush 146/300 Read block 61 len 40960 147/300 Read block 2315 len 32768 148/300 Write block 3936 len 40960 data: 2 3 3 4 3 2 2 3 2 2 149/300 Write block 4603 len 4096 data: 7 150/300 Flush 151/300 Write block 891 len 12288 data: 6 7 6 152/300 Read block 3254 len 36864 153/300 Read block 381 len 40960 154/300 Read block 3751 len 36864 155/300 Read block 2850 len 28672 156/300 Write block 1807 len 4096 data: 5 157/300 Read block 3387 len 8192 158/300 Read block 442 len 4096 159/300 Flush 160/300 Read block 785 len 40960 161/300 Read block 2414 len 16384 162/300 Read block 4581 len 20480 163/300 Read block 1004 len 32768 164/300 Write block 3606 len 4096 data: 7 165/300 Write block 4019 len 12288 data: 4 4 3 166/300 Read block 1429 len 36864 167/300 Flush 168/300 Write block 2596 len 36864 data: 3 5 5 6 4 3 3 4 4 169/300 Read block 1297 len 4096 170/300 Read block 2985 len 32768 171/300 Write block 4102 len 24576 data: 6 3 3 3 2 2 172/300 Write block 2484 len 32768 data: 5 4 5 6 7 6 6 6 173/300 Write block 212 len 32768 data: 5 5 5 5 6 5 5 4 174/300 Flush 175/300 Read block 699 len 24576 176/300 Write block 2542 len 20480 data: 5 5 6 6 7 177/300 Flush 178/300 Read block 2489 len 12288 179/300 Write block 3786 len 4096 data: 3 180/300 Read block 4478 len 28672 181/300 Flush 182/300 Write block 203 len 12288 data: 4 5 5 183/300 Read block 4940 len 16384 184/300 Flush 185/300 Flush 186/300 Write block 3406 len 16384 data: 3 3 3 4 187/300 Read block 744 len 12288 188/300 Flush 189/300 Read block 2018 len 40960 190/300 Write block 129 len 24576 data: 6 6 6 5 6 6 191/300 Read block 4907 len 40960 192/300 Read block 2168 len 8192 193/300 Write block 841 len 12288 data: 3 3 4 194/300 Read block 355 len 4096 195/300 Read block 1708 len 32768 196/300 Flush 197/300 Read block 4468 len 32768 198/300 Read block 790 len 32768 199/300 Read block 2185 len 4096 200/300 Write block 2745 len 8192 data: 3 5 201/300 Write block 4345 len 28672 data: 7 5 3 5 4 5 5 202/300 Read block 4348 len 20480 203/300 Flush 204/300 Write block 2632 len 36864 data: 6 6 6 9 9 9 8 8 8 205/300 Flush 206/300 Read block 4729 len 32768 207/300 Read block 1075 len 20480 208/300 Read block 4905 len 36864 209/300 Read block 4440 len 24576 210/300 Read block 1821 len 12288 211/300 Write block 4360 len 16384 data: 3 3 3 5 212/300 Write block 1436 len 8192 data: 3 3 213/300 Write block 3058 len 16384 data: 7 8 9 7 214/300 Read block 3491 len 24576 215/300 Read block 4178 len 24576 216/300 Read block 2979 len 20480 217/300 Read block 2312 len 20480 218/300 Flush 219/300 Read block 89 len 40960 220/300 Flush 221/300 Flush 222/300 Flush 223/300 Write block 308 len 40960 data: 5 7 8 10 9 9 8 8 8 7 224/300 Write block 536 len 32768 data: 9 8 7 7 7 7 6 5 225/300 Read block 2784 len 28672 226/300 Write block 3767 len 20480 data: 9 9 7 7 7 227/300 Flush 228/300 Write block 3537 len 32768 data: 7 7 7 8 9 9 9 8 229/300 Read block 2474 len 8192 230/300 Read block 2518 len 36864 231/300 Write block 2143 len 36864 data: 5 5 5 4 5 5 4 4 5 232/300 Read block 975 len 8192 233/300 Write block 3494 len 4096 data: 3 234/300 Flush 235/300 Read block 4901 len 36864 236/300 Write block 1538 len 36864 data: 2 2 2 2 3 3 4 4 4 237/300 Write block 4440 len 28672 data: 4 5 7 7 8 6 8 238/300 Write block 2873 len 12288 data: 5 5 4 239/300 Read block 4222 len 32768 240/300 Write block 3004 len 32768 data: 4 4 3 3 4 4 5 4 241/300 Read block 1376 len 36864 242/300 Write block 1440 len 8192 data: 2 3 243/300 Write block 3369 len 8192 data: 4 3 244/300 Read block 1738 len 28672 245/300 Write block 4429 len 24576 data: 5 4 4 6 6 7 246/300 Write block 1090 len 4096 data: 7 247/300 Read block 4832 len 4096 248/300 Write block 3678 len 40960 data: 5 4 5 5 4 3 4 5 3 3 249/300 Read block 4891 len 4096 250/300 Read block 510 len 8192 251/300 Write block 1115 len 40960 data: 8 6 6 6 5 5 6 5 4 4 252/300 Write block 1739 len 40960 data: 4 5 3 3 5 5 6 6 6 5 253/300 Write block 2715 len 16384 data: 4 6 5 4 254/300 Write block 688 len 8192 data: 7 7 255/300 Read block 3862 len 12288 256/300 Read block 3229 len 28672 257/300 Read block 1410 len 4096 258/300 Write block 1346 len 4096 data: 10 259/300 Write block 1472 len 8192 data: 4 3 260/300 Write block 1509 len 20480 data: 6 7 6 4 4 261/300 Read block 3181 len 24576 262/300 Read block 1656 len 32768 263/300 Read block 2084 len 32768 264/300 Read block 604 len 12288 265/300 Write block 683 len 24576 data: 6 6 6 5 6 8 266/300 Write block 20 len 36864 data: 6 4 4 5 5 6 5 6 6 267/300 Read block 4415 len 8192 268/300 Read block 2242 len 28672 269/300 Read block 2675 len 24576 270/300 Write block 2989 len 20480 data: 5 5 5 4 4 271/300 Write block 1964 len 4096 data: 5 272/300 Write block 4855 len 20480 data: 2 3 3 3 4 273/300 Read block 4962 len 36864 274/300 Write block 4942 len 16384 data: 4 4 3 3 275/300 Write block 1971 len 24576 data: 5 5 4 4 3 2 276/300 Read block 2568 len 8192 277/300 Flush 278/300 Read block 4532 len 40960 279/300 Write block 150 len 32768 data: 6 5 5 5 4 5 3 3 280/300 Write block 3813 len 16384 data: 6 6 6 6 281/300 Read block 3006 len 24576 282/300 Read block 3367 len 8192 283/300 Read block 1089 len 40960 284/300 Read block 4227 len 28672 285/300 Read block 358 len 32768 286/300 Read block 2597 len 40960 287/300 Read block 4453 len 24576 288/300 Write block 2605 len 4096 data: 5 289/300 Write block 2957 len 8192 data: 4 4 290/300 Write block 4860 len 12288 data: 4 5 3 291/300 Write block 4269 len 32768 data: 13 11 8 8 5 4 3 3 292/300 Write block 2439 len 40960 data: 5 4 4 4 5 5 6 8 8 6 293/300 Read block 1087 len 20480 294/300 Flush 295/300 Read block 1127 len 4096 296/300 Read block 3941 len 4096 297/300 Read block 2910 len 24576 298/300 Write block 3917 len 32768 data: 3 3 3 3 2 3 5 7 299/300 Flush 300/300 Write block 3784 len 36864 data: 2 2 4 4 7 8 7 8 9 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:35:16.482 INFO request completed, latency_us: 293, response_code: 204, uri: /start/cid/2, method: POST, req_id: cdec1bdc-bb79-42c8-9490-afa45ca73fb4, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 6946 Acked 7945 Write 8 Done Done New true 6947 Acked 7946 Read 1 Done Done New true 6948 Acked 7947 Write 2 Done Done New false 6949 Acked 7948 Read 7 Done Done New false 6950 Acked 7949 Write 2 Done Done New false 6951 Acked 7950 Read 8 Done Done New false 6952 Acked 7951 Flush 0 Done Done New false 6953 Acked 7952 Flush 0 Done Done New false 6954 Acked 7953 Write 10 Done Done New false 6955 Acked 7954 Write 1 Done Done New false 6956 Acked 7955 Write 4 Done Done New false 6957 Acked 7956 Write 1 Done Done New false 6958 Acked 7957 Read 7 Done Done New false 6959 Acked 7958 Write 5 Done Done New false 6960 Acked 7959 Flush 0 Done Done New false 6961 Acked 7960 Read 9 Done Done New false 6962 Acked 7961 Read 7 Done Done New false 6963 Acked 7962 Read 1 Done Done New false 6964 Acked 7963 Read 1 Done Done New false 6965 Acked 7964 Flush 0 Done Done New false 6966 Acked 7965 Write 4 Done Done New false 6967 Acked 7966 Read 1 Done Done New false 6968 Acked 7967 Read 5 Done Done New false 6969 Acked 7968 Write 4 Done Done New false 6970 Acked 7969 Write 3 Done Done New false 6971 Acked 7970 Flush 0 Done Done New false 6972 Acked 7971 Write 8 Done Done New false 6973 Acked 7972 Write 2 Done Done New false 6974 Acked 7973 Read 10 Done Done New false 6975 Acked 7974 Read 5 Done Done New false 6976 Acked 7975 Read 1 Done Done New false 6977 Acked 7976 Read 1 Done Done New false 6978 Acked 7977 Read 7 Done Done New false 6979 Acked 7978 Write 5 Done Done New false 6980 Acked 7979 Read 6 Done Done New false 6981 Acked 7980 Write 7 Done Done New false 6982 Acked 7981 Read 4 Done Done New false 6983 Acked 7982 Write 2 Done Done New false 6984 Acked 7983 Read 1 Done Done New false 6985 Acked 7984 Write 7 Done Done New false 6986 Acked 7985 Flush 0 Done Done New false 6987 Acked 7986 Read 5 Done Done New false 6988 Acked 7987 Read 7 Done Done New false 6989 Acked 7988 Write 7 Done Done New false 6990 Acked 7989 Read 9 Done Done New false 6991 Acked 7990 Read 2 Done Done New false 6992 Acked 7991 Read 1 Done Done New false 6993 Acked 7992 Read 9 Done Done New false 6994 Acked 7993 Read 4 Done Done New false 6995 Acked 7994 Write 6 Done Done New false 6996 Acked 7995 Write 5 Done Done New false 6997 Acked 7996 Write 1 Done Done New false 6998 Acked 7997 Read 10 Done Done New false 6999 Acked 7998 Write 2 Done Done New false 7000 Acked 7999 Read 9 Done Done New false 7001 Acked 8000 Write 7 Done Done New false 7002 Acked 8001 Read 10 Done Done New false 7003 Acked 8002 Read 8 Done Done New false 7004 Acked 8003 Write 7 Done Done New false 7005 Acked 8004 Write 10 Done Done New false 7006 Acked 8005 Write 5 Done Done New false 7007 Acked 8006 Read 1 Done Done New false 7008 Acked 8007 Write 5 Done Done New false 7009 Acked 8008 Read 9 Done Done New false 7010 Acked 8009 Read 3 Done Done New false 7011 Acked 8010 Write 2 Done Done New false 7012 Acked 8011 Read 10 Done Done New false 7013 Acked 8012 Read 5 Done Done New false 7014 Acked 8013 Read 7 Done Done New false 7015 Acked 8014 Read 3 Done Done New false 7016 Acked 8015 Write 1 Done Done New false 7017 Acked 8016 Write 7 Done Done New false 7018 Acked 8017 Read 2 Done Done New false 7019 Acked 8018 Read 2 Done Done New false 7020 Acked 8019 Read 8 Done Done New false 7021 Acked 8020 Read 5 Done Done New false 7022 Acked 8021 Read 9 Done Done New false 7023 Acked 8022 Write 7 Done Done New false 7024 Acked 8023 Write 4 Done Done New false 7025 Acked 8024 Read 9 Done Done New false 7026 Acked 8025 Write 4 Done Done New false 7027 Acked 8026 Read 5 Done Done New false 7028 Acked 8027 Write 6 Done Done New false 7029 Acked 8028 Write 5 Done Done New false 7030 Acked 8029 Read 5 Done Done New false 7031 Acked 8030 Read 9 Done Done New false 7032 Acked 8031 Read 7 Done Done New false 7033 Acked 8032 Flush 0 Done Done New false 7034 Acked 8033 Read 4 Done Done New false 7035 Acked 8034 Read 10 Done Done New false 7036 Acked 8035 Flush 0 Done Done New false 7037 Acked 8036 Read 5 Done Done New false 7038 Acked 8037 Write 10 Done Done New false 7039 Acked 8038 Read 6 Done Done New false 7040 Acked 8039 Read 7 Done Done New false 7041 Acked 8040 Read 5 Done Done New false 7042 Acked 8041 Write 9 Done Done New false 7043 Acked 8042 Read 4 Done Done New false 7044 Acked 8043 Write 6 Done Done New false 7045 Acked 8044 Write 10 Done Done New false 7046 Acked 8045 Flush 0 Done Done New false 7047 Acked 8046 Flush 0 Done Done New false 7048 Acked 8047 Write 10 Done Done New false 7049 Acked 8048 Read 9 Done Done New false 7050 Acked 8049 Read 3 Done Done New false 7051 Acked 8050 Write 2 Done Done New false 7052 Acked 8051 Read 8 Done Done New false 7053 Acked 8052 Write 9 Done Done New false 7054 Acked 8053 Read 5 Done Done New false 7055 Acked 8054 Write 4 Done Done New false 7056 Acked 8055 Read 7 Done Done New false 7057 Acked 8056 Flush 0 Done Done New false 7058 Acked 8057 Flush 0 Done Done New false 7059 Acked 8058 Flush 0 Done Done New false 7060 Acked 8059 Flush 0 Done Done New false 7061 Acked 8060 Flush 0 Done Done New false 7062 Acked 8061 Read 4 Done Done New false 7063 Acked 8062 Flush 0 Done Done New false 7064 Acked 8063 Read 5 Done Done New false 7065 Acked 8064 Read 3 Done Done New false 7066 Acked 8065 Read 1 Done Done New false 7067 Acked 8066 Read 1 Done Done New false 7068 Acked 8067 Write 3 Done Done New false 7069 Acked 8068 Write 9 Done Done New false 7070 Acked 8069 Read 7 Done Done New false 7071 Acked 8070 Read 4 Done Done New false 7072 Acked 8071 Write 5 Done Done New false 7073 Acked 8072 Write 3 Done Done New false 7074 Acked 8073 Read 9 Done Done New false 7075 Acked 8074 Read 5 Done Done New false 7076 Acked 8075 Read 4 Done Done New false 7077 Acked 8076 Write 1 Done Done New false 7078 Acked 8077 Write 2 Done Done New false 7079 Acked 8078 Read 7 Done Done New false 7080 Acked 8079 Write 2 Done Done New false 7081 Acked 8080 Flush 0 Done Done New false 7082 Acked 8081 Read 3 Done Done New false 7083 Acked 8082 Flush 0 Done Done New false 7084 Acked 8083 Write 7 Done Done New false 7085 Acked 8084 Write 1 Done Done New false 7086 Acked 8085 Write 4 Done Done New false 7087 Acked 8086 Flush 0 Done Done New false 7088 Acked 8087 Read 9 Done Done New false 7089 Acked 8088 Write 5 Done Done New false 7090 Acked 8089 Write 9 Done Done New false 7091 Acked 8090 Flush 0 Done Done New false 7092 Acked 8091 Read 10 Done Done New false 7093 Acked 8092 Read 8 Done Done New false 7094 Acked 8093 Write 10 Done Done New false 7095 Acked 8094 Write 1 Done Done New false 7096 Acked 8095 Flush 0 Done Done New false 7097 Acked 8096 Write 3 Done Done New false 7098 Acked 8097 Read 9 Done Done New false 7099 Acked 8098 Read 10 Done Done New false 7100 Acked 8099 Read 9 Done Done New false 7101 Acked 8100 Read 7 Done Done New false 7102 Acked 8101 Write 1 Done Done New false 7103 Acked 8102 Read 2 Done Done New false 7104 Acked 8103 Read 1 Done Done New false 7105 Acked 8104 Flush 0 Done Done New false 7106 Acked 8105 Read 10 Done Done New false 7107 Acked 8106 Read 4 Done Done New false 7108 Acked 8107 Read 5 Done Done New false 7109 Acked 8108 Read 8 Done Done New false 7110 Acked 8109 Write 1 Done Done New false 7111 Acked 8110 Write 3 Done Done New false 7112 Acked 8111 Read 9 Done Done New false 7113 Acked 8112 Flush 0 Done Done New false 7114 Acked 8113 Write 9 Done Done New false 7115 Acked 8114 Read 1 Done Done New false 7116 Acked 8115 Read 8 Done Done New false 7117 Acked 8116 Write 6 Done Done New false 7118 Acked 8117 Write 8 Done Done New false 7119 Acked 8118 Write 8 Done Done New false 7120 Acked 8119 Flush 0 Done Done New false 7121 Acked 8120 Read 6 Done Done New false 7122 Acked 8121 Write 5 Done Done New false 7123 Acked 8122 Flush 0 Done Done New false 7124 Acked 8123 Read 3 Done Done New false 7125 Acked 8124 Write 1 Done Done New false 7126 Acked 8125 Read 7 Done Done New false 7127 Acked 8126 Flush 0 Done Done New false 7128 Acked 8127 Write 3 Done Done New false 7129 Acked 8128 Read 4 Done Done New false 7130 Acked 8129 Flush 0 Done Done New false 7131 Acked 8130 Flush 0 Done Done New false 7132 Acked 8131 Write 4 Done Done New false 7133 Acked 8132 Read 3 Done Done New false 7134 Acked 8133 Flush 0 Done Done New false 7135 Acked 8134 Read 10 Done Done New false 7136 Acked 8135 Write 6 Done Done New false 7137 Acked 8136 Read 10 Done Done New false 7138 Acked 8137 Read 2 Done Done New false 7139 Acked 8138 Write 3 Done Done New false 7140 Acked 8139 Read 1 Done Done New false 7141 Acked 8140 Read 8 Done Done New false 7142 Acked 8141 Flush 0 Done Done New false 7143 Acked 8142 Read 8 Done Done New false 7144 Acked 8143 Read 8 Done Done New false 7145 Acked 8144 Read 1 Done Done New false 7146 Acked 8145 Write 2 Done Done New false 7147 Acked 8146 Write 7 Done Done New false 7148 Acked 8147 Read 5 Done Done New false 7149 Acked 8148 Flush 0 Done Done New false 7150 Acked 8149 Write 9 Done Done New false 7151 Acked 8150 Flush 0 Done Done New false 7152 Acked 8151 Read 8 Done Done New false 7153 Acked 8152 Read 5 Done Done New false 7154 Acked 8153 Read 9 Done Done New false 7155 Acked 8154 Read 6 Done Done New false 7156 Acked 8155 Read 3 Done Done New false 7157 Acked 8156 Write 4 Done Done New false 7158 Acked 8157 Write 2 Done Done New false 7159 Acked 8158 Write 4 Done Done New false 7160 Acked 8159 Read 6 Done Done New false 7161 Acked 8160 Read 6 Done Done New false 7162 Acked 8161 Read 5 Done Done New false 7163 Acked 8162 Read 5 Done Done New false 7164 Acked 8163 Flush 0 Done Done New false 7165 Acked 8164 Read 10 Done Done New false 7166 Acked 8165 Flush 0 Done Done New false 7167 Acked 8166 Flush 0 Done Done New false 7168 Acked 8167 Flush 0 Done Done New false 7169 Acked 8168 Write 10 Done Done New false 7170 Acked 8169 Write 8 Done Done New false 7171 Acked 8170 Read 7 Done Done New false 7172 Acked 8171 Write 5 Done Done New false 7173 Acked 8172 Flush 0 Done Done New false 7174 Acked 8173 Write 8 Done Done New false 7175 Acked 8174 Read 2 Done Done New false 7176 Acked 8175 Read 9 Done Done New false 7177 Acked 8176 Write 9 Done Done New false 7178 Acked 8177 Read 2 Done Done New false 7179 Acked 8178 Write 1 Done Done New false 7180 Acked 8179 Flush 0 Done Done New false 7181 Acked 8180 Read 9 Done Done New false 7182 Acked 8181 Write 9 Done Done New false 7183 Acked 8182 Write 7 Done Done New false 7184 Acked 8183 Write 3 Done Done New false 7185 Acked 8184 Read 8 Done Done New false 7186 Acked 8185 Write 8 Done Done New false 7187 Acked 8186 Read 9 Done Done New false 7188 Acked 8187 Write 2 Done Done New false 7189 Acked 8188 Write 2 Done Done New false 7190 Acked 8189 Read 7 Done Done New false 7191 Acked 8190 Write 6 Done Done New false 7192 Acked 8191 Write 1 Done Done New false 7193 Acked 8192 Read 1 Done Done New false 7194 Acked 8193 Write 10 Done Done New false 7195 Acked 8194 Read 1 Done Done New false 7196 Acked 8195 Read 2 Done Done New false 7197 Acked 8196 Write 10 Done Done New false 7198 Acked 8197 Write 10 Done Done New false 7199 Acked 8198 Write 4 Done Done New false 7200 Acked 8199 Write 2 Done Done New false 7201 Acked 8200 Read 3 Done Done New false 7202 Acked 8201 Read 7 Done Done New false 7203 Acked 8202 Read 1 Done Done New false 7204 Acked 8203 Write 1 Done Done New false 7205 Acked 8204 Write 2 Done Done New false 7206 Acked 8205 Write 5 Done Done New false 7207 Acked 8206 Read 6 Done Done New false 7208 Acked 8207 Flush 0 Done Done New false 7209 Acked 8208 Read 8 Done Done New false 7210 Acked 8209 Read 8 Done Done New false 7211 Acked 8210 Read 3 Done Done New false 7212 Acked 8211 Write 6 Done Done New false 7213 Acked 8212 Write 9 Done Done New false 7214 Acked 8213 Read 2 Done Done New false 7215 Acked 8214 Read 7 Done Done New false 7216 Acked 8215 Read 6 Done Done New false 7217 Acked 8216 Write 5 Done Done New false 7218 Acked 8217 Write 1 Done Done New false 7219 Acked 8218 Write 5 Done Done New false 7220 Acked 8219 Read 9 Done Done New false 7221 Acked 8220 Write 4 Done Done New false 7222 Acked 8221 Write 6 Done Done New false 7223 Acked 8222 Read 2 Done Done New false 7224 Acked 8223 Flush 0 Done Done New false 7225 Acked 8224 Read 10 Done Done New false 7226 Acked 8225 Write 8 Done Done New false 7227 Acked 8226 Write 4 Done Done New false 7228 Acked 8227 Read 6 Done Done New false 7229 Acked 8228 Read 2 Done Done New false 7230 Acked 8229 Read 10 Done Done New false 7231 Acked 8230 Read 7 Done Done New false 7232 Acked 8231 Read 8 Done Done New false 7233 Acked 8232 Read 10 Done Done New false 7234 Acked 8233 Read 6 Done Done New false 7235 Acked 8234 Write 1 Done Done New false 7236 Acked 8235 Write 2 Done Done New false 7237 Acked 8236 Write 3 Done Done New false 7238 Acked 8237 Write 8 Done Done New false 7239 Acked 8238 Write 10 Done Done New false 7240 Acked 8239 Read 5 Done Done New false 7241 Acked 8240 Flush 0 Done Done New false 7242 Acked 8241 Read 1 Done Done New false 7243 Acked 8242 Read 1 Done Done New false 7244 Acked 8243 Read 6 Done Done New false 7245 Acked 8244 Write 8 Done Done New false 7246 Acked 8245 Flush 0 Done Done New false 7247 Acked 8246 Write 9 Done Done New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 302 302 Sent 0 0 0 0 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8245 8245 7944 Downstairs last five completed: 7944 7943 7942 7941 7940 Upstairs last five completed: 7247 7246 7245 7244 7243 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:16.888480675Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:16.888551536Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:16.888580384Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:16.888604639Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:16.888630946Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:16.888662772Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:16.888686541Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:16.888941657Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:17.889761987Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:17.889844807Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:17.88987362Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:17.889900649Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:17.889921375Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 303 jobs since flush 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:17.889955201Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:17.890213429Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1209) [8830][2] reports Running {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.89123381Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.891379926Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893337245Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893381688Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893406707Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 7944","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893680415Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893714498Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893737966Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893760327Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:18.893782667Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 8247 8246 8245 8244 8243 Upstairs last five completed: 7248 7247 7246 7245 7244 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 Jan 02 22:35:20.486 INFO request completed, latency_us: [8810] Got stop action so:false kr:false 452, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 06e864fc-423c-40b4-ad99-44a236d75514, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:35:20.487 INFO request completed, latency_us: 382, response_code: 200, uri: /state/cid/0, method: GET, req_id: ac4b74e7-8889-4b6c-9409-d3d7b9a8ca0d, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:20.48752089Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:20.487613849Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:20.487654367Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:20.487693437Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:20.48773755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:20.487789514Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:21.488858088Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:21.488928131Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:21.488956299Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:21.488980698Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:21.489006755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:21.489027276Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:21.489049526Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:21.489071707Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:22.490258761Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:22.490327725Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:22.490355858Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:22.490382331Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:22.490413582Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:22.490441592Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:22.490487431Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:23.491428178Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:23.491500409Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:23.49152841Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:23.491555263Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:23.491575709Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:23.491619773Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:23.491720602Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:35:24.489 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/0, method: GET, req_id: b1449dba-e603-46d9-ab7b-6d82356c7ad5, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 3237 len 16384 data: 8 7 8 9 002/300 Read block 3287 len 20480 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:24.492160906Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:24.492227553Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:24.492255992Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:24.492290229Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:24.492311804Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 2 jobs since flush 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:24.492334348Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:24.493692667Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 2630 len 36864 004/300 Flush 005/300 Write block 3802 len 36864 data: 5 6 6 5 4 3 3 2 3 006/300 Write block 528 len 28672 data: 6 6 5 5 6 5 7 007/300 Read block 1747 len 16384 008/300 Read block 4878 len 8192 009/300 Read block 4688 len 8192 010/300 Read block 4491 len 12288 011/300 Read block 4858 len 36864 012/300 Read block 1756 len 32768 013/300 Read block 4846 len 32768 014/300 Write block 109 len 8192 data: 4 4 015/300 Write block 2079 len 16384 data: 6 4 3 3 016/300 Read block 4408 len 40960 017/300 Write block 162 len 12288 data: 4 3 4 018/300 Flush 019/300 Write block 465 len 36864 data: 4 3 3 3 5 6 5 6 6 020/300 Read block 473 len 24576 021/300 Write block 1275 len 28672 data: 6 7 7 5 4 4 5 022/300 Flush 023/300 Read block 3326 len 24576 024/300 Read block 119 len 20480 025/300 Read block 1567 len 8192 026/300 Flush 027/300 Read block 566 len 4096 028/300 Write block 1129 len 24576 data: 7 6 6 7 7 9 029/300 Read block 818 len 16384 030/300 Write block 2203 len 24576 data: 4 4 4 3 3 4 031/300 Write block 4356 len 28672 data: 2 2 2 3 4 4 4 032/300 Flush 033/300 Write block 4249 len 16384 data: 9 8 7 7 034/300 Read block 4656 len 24576 035/300 Write block 1270 len 4096 data: 6 036/300 Read block 2341 len 4096 037/300 Read block 4457 len 40960 038/300 Flush 039/300 Write block 422 len 8192 data: 8 5 040/300 Write block 253 len 40960 data: 3 4 4 5 4 4 4 4 5 5 041/300 Write block 2529 len 28672 data: 5 5 5 7 7 6 6 042/300 Write block 694 len 16384 data: 5 5 6 4 043/300 Read block 1901 len 16384 044/300 Read block 650 len 20480 045/300 Read block 2284 len 4096 046/300 Write block 770 len 28672 data: 2 3 4 4 5 4 3 047/300 Read block 2171 len 16384 048/300 Read block 4728 len 4096 049/300 Write block 955 len 12288 data: 3 4 4 050/300 Write block 2732 len 16384 data: 5 8 6 6 051/300 Write block 572 len 36864 data: 4 5 4 4 5 6 6 6 6 052/300 Write block 1064 len 36864 data: 4 5 6 6 6 7 9 8 11 053/300 Write block 1321 len 40960 data: 5 7 7 6 6 6 7 6 7 6 054/300 Flush 055/300 Read block 4170 len 16384 056/300 Read block 4363 len 4096 057/300 Write block 4321 len 32768 data: 7 7 7 7 7 8 8 6 058/300 Read block 3188 len 24576 059/300 Read block 3885 len 40960 060/300 Write block 1873 len 40960 data: 6 7 7 5 5 6 5 5 6 7 061/300 Write block 4923 len 28672 data: 7 9 10 11 11 10 9 062/300 Read block 1697 len 36864 063/300 Read block 4974 len 32768 064/300 Read block 3782 len 24576 065/300 Read block 2623 len 12288 066/300 Write block 2476 len 16384 data: 7 7 4 6 067/300 Write block 2596 len 24576 data: 4 6 6 7 5 4 068/300 Read block 4845 len 36864 069/300 Read block 4059 len 12288 070/300 Flush 071/300 Write block 1653 len 24576 data: 6 5 7 7 7 6 072/300 Write block 339 len 20480 data: 7 6 6 6 5 073/300 Read block 3671 len 16384 074/300 Read block 2357 len 36864 075/300 Write block 2763 len 32768 data: 6 7 8 7 7 7 9 10 076/300 Write block 4092 len 16384 data: 8 6 7 6 077/300 Read block 4078 len 40960 078/300 Read block 1235 len 12288 079/300 Read block 3926 len 32768 080/300 Flush 081/300 Write block 1041 len 24576 data: 5 3 3 3 3 4 082/300 Flush 083/300 Read block 3086 len 4096 084/300 Read block 51 len 20480 085/300 Read block 4156 len 28672 086/300 Write block 3028 len 12288 data: 6 5 4 087/300 Read block 277 len 28672 088/300 Write block 2188 len 40960 data: 5 6 5 4 5 4 5 7 9 8 089/300 Read block 1799 len 12288 090/300 Write block 3943 len 12288 data: 4 3 3 091/300 Flush 092/300 Read block 4194 len 8192 093/300 Read block 2377 len 28672 094/300 Write block 923 len 4096 data: 5 095/300 Read block 1320 len 32768 096/300 Read block 1901 len 28672 097/300 Write block 3942 len 12288 data: 3 5 4 098/300 Write block 3233 len 8192 data: 7 7 099/300 Read block 90 len 36864 100/300 Read block 3301 len 36864 101/300 Write block 1276 len 20480 data: 8 8 6 5 5 102/300 Read block 4924 len 40960 103/300 Write block 3390 len 20480 data: 4 5 4 4 4 104/300 Write block 2561 len 24576 data: 3 4 5 5 5 4 105/300 Read block 4588 len 36864 106/300 Read block 4697 len 16384 107/300 Write block 2182 len 8192 data: 4 4 108/300 Write block 2331 len 40960 data: 7 7 5 6 6 6 7 6 5 6 109/300 Write block 1831 len 20480 data: 4 6 4 4 3 110/300 Read block 1525 len 8192 111/300 Write block 4775 len 20480 data: 6 5 4 5 5 112/300 Write block 297 len 12288 data: 8 6 4 113/300 Read block 3890 len 36864 114/300 Write block 4119 len 40960 data: 5 5 5 4 4 5 6 6 8 6 115/300 Write block 4411 len 40960 data: 2 2 3 3 3 3 4 6 7 6 116/300 Write block 4115 len 8192 data: 3 3 117/300 Read block 3637 len 24576 118/300 Write block 4965 len 40960 data: 8 5 5 5 6 6 5 3 2 3 119/300 Read block 3723 len 4096 120/300 Write block 3747 len 16384 data: 4 5 3 3 121/300 Write block 4620 len 24576 data: 5 5 5 5 4 5 122/300 Write block 1200 len 32768 data: 4 4 4 4 5 5 7 5 123/300 Write block 3409 len 28672 data: 5 4 4 6 5 5 5 124/300 Read block 1896 len 8192 125/300 Flush 126/300 Write block 3023 len 40960 data: 3 2 4 4 6 7 6 5 6 6 127/300 Flush 128/300 Write block 3372 len 36864 data: 2 2 3 3 3 4 5 5 5 129/300 Write block 213 len 8192 data: 6 6 130/300 Read block 3379 len 20480 131/300 Write block 220 len 28672 data: 4 5 6 6 7 7 6 132/300 Write block 4553 len 8192 data: 4 4 133/300 Read block 2052 len 12288 134/300 Flush 135/300 Write block 819 len 24576 data: 5 4 3 5 5 5 136/300 Write block 2030 len 12288 data: 8 9 10 137/300 Flush 138/300 Read block 3349 len 12288 139/300 Read block 925 len 36864 140/300 Read block 2281 len 24576 141/300 Read block 3151 len 28672 142/300 Read block 3121 len 20480 143/300 Read block 2917 len 24576 144/300 Write block 93 len 8192 data: 6 7 145/300 Write block 4770 len 40960 data: 6 7 8 7 6 7 6 5 6 6 146/300 Write block 4531 len 24576 data: 2 2 3 5 6 6 147/300 Read block 4509 len 20480 148/300 Write block 628 len 28672 data: 6 5 4 4 4 5 6 149/300 Read block 1520 len 8192 150/300 Write block 3044 len 24576 data: 4 4 5 6 8 5 151/300 Read block 1545 len 20480 152/300 Read block 1556 len 8192 153/300 Write block 2664 len 16384 data: 3 3 2 2 154/300 Flush 155/300 Read block 807 len 32768 156/300 Write block 3877 len 40960 data: 4 4 4 4 7 8 9 7 8 9 157/300 Read block 315 len 32768 158/300 Read block 1983 len 40960 159/300 Read block 3644 len 32768 160/300 Write block 3857 len 40960 data: 6 5 5 5 6 6 6 6 4 4 161/300 Write block 2289 len 12288 data: 4 3 2 162/300 Write block 740 len 12288 data: 3 3 3 163/300 Read block 3415 len 20480 164/300 Read block 3829 len 28672 165/300 Read block 4281 len 32768 166/300 Read block 451 len 4096 167/300 Read block 46 len 4096 168/300 Read block 2089 len 8192 169/300 Flush 170/300 Write block 4356 len 40960 data: 3 3 3 4 5 5 5 6 5 5 171/300 Write block 2035 len 8192 data: 7 8 172/300 Read block 3991 len 28672 173/300 Read block 799 len 28672 174/300 Flush 175/300 Read block 2584 len 12288 176/300 Write block 4647 len 12288 data: 5 6 5 177/300 Flush 178/300 Write block 1788 len 40960 data: 4 5 4 4 4 3 4 4 4 3 179/300 Read block 2968 len 24576 180/300 Read block 1875 len 36864 181/300 Write block 1093 len 24576 data: 7 8 7 5 4 5 182/300 Write block 2020 len 28672 data: 4 4 4 4 2 2 2 183/300 Write block 4367 len 32768 data: 4 3 3 3 3 3 3 3 184/300 Read block 4533 len 12288 185/300 Write block 4010 len 20480 data: 10 8 7 7 8 186/300 Write block 1456 len 32768 data: 3 5 5 5 6 7 5 7 187/300 Read block 1102 len 32768 188/300 Read block 2647 len 24576 189/300 Read block 2409 len 4096 190/300 Read block 4706 len 20480 191/300 Write block 4099 len 24576 data: 4 6 6 7 4 4 192/300 Read block 898 len 20480 193/300 Write block 1561 len 8192 data: 5 3 194/300 Read block 108 len 20480 195/300 Write block 415 len 32768 data: 3 5 5 7 7 7 8 9 196/300 Write block 1502 len 36864 data: 5 7 8 7 8 7 7 7 8 197/300 Write block 4955 len 4096 data: 4 198/300 Flush 199/300 Write block 1732 len 36864 data: 4 3 3 3 5 4 5 5 6 200/300 Read block 4436 len 8192 201/300 Write block 2849 len 40960 data: 6 5 6 4 4 6 6 6 7 7 202/300 Read block 3613 len 40960 203/300 Read block 2049 len 24576 204/300 Read block 1207 len 36864 205/300 Write block 1003 len 4096 data: 3 206/300 Read block 3291 len 32768 207/300 Read block 1814 len 20480 208/300 Read block 3973 len 36864 209/300 Write block 2913 len 36864 data: 7 5 4 5 6 4 4 5 5 210/300 Read block 1919 len 4096 211/300 Read block 2715 len 20480 212/300 Read block 3134 len 40960 213/300 Read block 4321 len 24576 214/300 Flush 215/300 Write block 3776 len 28672 data: 4 5 6 6 5 3 3 216/300 Write block 1725 len 8192 data: 3 3 217/300 Read block 394 len 12288 218/300 Read block 379 len 12288 219/300 Read block 4487 len 8192 220/300 Read block 4393 len 40960 221/300 Read block 598 len 8192 222/300 Write block 2648 len 8192 data: 5 5 223/300 Read block 4729 len 4096 224/300 Write block 1134 len 36864 data: 10 8 8 9 7 6 5 7 6 225/300 Flush 226/300 Write block 3308 len 28672 data: 5 6 6 7 4 3 3 227/300 Read block 1284 len 28672 228/300 Write block 3760 len 12288 data: 5 6 6 229/300 Write block 3465 len 24576 data: 4 4 4 5 4 6 230/300 Read block 1734 len 12288 231/300 Flush 232/300 Read block 4968 len 4096 233/300 Read block 614 len 4096 234/300 Write block 1520 len 36864 data: 5 5 5 4 4 4 6 6 2 235/300 Read block 3953 len 12288 236/300 Write block 2148 len 20480 data: 6 5 5 6 7 237/300 Read block 2880 len 40960 238/300 Flush 239/300 Write block 4974 len 12288 data: 4 4 5 240/300 Write block 3391 len 32768 data: 6 5 5 5 4 6 7 4 241/300 Read block 3666 len 32768 242/300 Read block 3154 len 16384 243/300 Write block 2211 len 16384 data: 5 4 4 3 244/300 Read block 4147 len 28672 245/300 Write block 1190 len 16384 data: 6 6 6 5 246/300 Read block 1820 len 32768 247/300 Read block 828 len 20480 248/300 Write block 1484 len 12288 data: 4 4 3 249/300 Read block 1239 len 16384 250/300 Write block 140 len 24576 data: 5 5 4 4 5 5 251/300 Write block 4852 len 4096 data: 2 252/300 Read block 3398 len 8192 253/300 Write block 90 len 28672 data: 5 5 5 7 8 6 6 254/300 Write block 1654 len 12288 data: 6 8 8 255/300 Read block 1822 len 24576 256/300 Write block 2109 len 32768 data: 4 5 4 3 3 3 3 5 257/300 Read block 2694 len 20480 258/300 Write block 2471 len 4096 data: 5 259/300 Read block 221 len 40960 260/300 Write block 3888 len 8192 data: 8 6 261/300 Write block 3358 len 20480 data: 7 9 9 7 7 262/300 Write block 3650 len 36864 data: 6 5 3 4 5 5 4 5 5 263/300 Write block 2805 len 16384 data: 4 5 4 4 264/300 Write block 4663 len 40960 data: 4 4 5 5 4 3 3 3 4 4 265/300 Flush 266/300 Read block 3921 len 4096 267/300 Write block 3691 len 8192 data: 5 5 268/300 Write block 535 len 16384 data: 10 10 9 8 269/300 Write block 4923 len 20480 data: 8 10 11 12 12 270/300 Read block 15 len 16384 271/300 Read block 1287 len 8192 272/300 Read block 3658 len 28672 273/300 Read block 3616 len 28672 274/300 Read block 1905 len 12288 275/300 Write block 4045 len 24576 data: 8 8 9 8 7 7 276/300 Read block 4540 len 8192 277/300 Write block 1967 len 36864 data: 4 4 5 5 6 6 5 5 4 278/300 Write block 4740 len 40960 data: 9 8 7 6 5 5 5 6 6 7 279/300 Read block 818 len 4096 280/300 Read block 4756 len 8192 281/300 Flush 282/300 Read block 2362 len 8192 283/300 Read block 4906 len 36864 284/300 Read block 926 len 24576 285/300 Flush 286/300 Read block 3347 len 4096 287/300 Write block 4241 len 16384 data: 5 4 4 5 288/300 Flush 289/300 Read block 97 len 28672 290/300 Write block 3607 len 32768 data: 6 7 7 7 8 5 5 5 291/300 Write block 752 len 28672 data: 2 2 3 3 5 5 5 292/300 Write block 930 len 20480 data: 5 4 6 5 6 293/300 Write block 955 len 8192 data: 4 5 294/300 Flush 295/300 Write block 1275 len 36864 data: 7 9 9 7 6 6 6 5 6 296/300 Read block 1602 len 40960 297/300 Read block 1753 len 36864 298/300 Read block 1774 len 8192 299/300 Flush 300/300 Write block 2693 len 16384 data: 6 5 4 6 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:35:25.084 INFO request completed, latency_us: 334, response_code: 204, uri: /start/cid/0, method: POST, req_id: b5209282-92aa-4b60-8cb2-8861ee8c8981, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 7249 Acked 8248 Write 4 New Done Done true 7250 Acked 8249 Read 5 New Done Done true 7251 Acked 8250 Read 9 New Done Done false 7252 Acked 8251 Flush 0 New Done Done false 7253 Acked 8252 Write 9 New Done Done false 7254 Acked 8253 Write 7 New Done Done false 7255 Acked 8254 Read 4 New Done Done false 7256 Acked 8255 Read 2 New Done Done false 7257 Acked 8256 Read 2 New Done Done false 7258 Acked 8257 Read 3 New Done Done false 7259 Acked 8258 Read 9 New Done Done false 7260 Acked 8259 Read 8 New Done Done false 7261 Acked 8260 Read 8 New Done Done false 7262 Acked 8261 Write 2 New Done Done false 7263 Acked 8262 Write 4 New Done Done false 7264 Acked 8263 Read 10 New Done Done false 7265 Acked 8264 Write 3 New Done Done false 7266 Acked 8265 Flush 0 New Done Done false 7267 Acked 8266 Write 9 New Done Done false 7268 Acked 8267 Read 6 New Done Done false 7269 Acked 8268 Write 7 New Done Done false 7270 Acked 8269 Flush 0 New Done Done false 7271 Acked 8270 Read 6 New Done Done false 7272 Acked 8271 Read 5 New Done Done false 7273 Acked 8272 Read 2 New Done Done false 7274 Acked 8273 Flush 0 New Done Done false 7275 Acked 8274 Read 1 New Done Done false 7276 Acked 8275 Write 6 New Done Done false 7277 Acked 8276 Read 4 New Done Done false 7278 Acked 8277 Write 6 New Done Done false 7279 Acked 8278 Write 7 New Done Done false 7280 Acked 8279 Flush 0 New Done Done false 7281 Acked 8280 Write 4 New Done Done false 7282 Acked 8281 Read 6 New Done Done false 7283 Acked 8282 Write 1 New Done Done false 7284 Acked 8283 Read 1 New Done Done false 7285 Acked 8284 Read 10 New Done Done false 7286 Acked 8285 Flush 0 New Done Done false 7287 Acked 8286 Write 2 New Done Done false 7288 Acked 8287 Write 10 New Done Done false 7289 Acked 8288 Write 7 New Done Done false 7290 Acked 8289 Write 4 New Done Done false 7291 Acked 8290 Read 4 New Done Done false 7292 Acked 8291 Read 5 New Done Done false 7293 Acked 8292 Read 1 New Done Done false 7294 Acked 8293 Write 7 New Done Done false 7295 Acked 8294 Read 4 New Done Done false 7296 Acked 8295 Read 1 New Done Done false 7297 Acked 8296 Write 3 New Done Done false 7298 Acked 8297 Write 4 New Done Done false 7299 Acked 8298 Write 9 New Done Done false 7300 Acked 8299 Write 9 New Done Done false 7301 Acked 8300 Write 10 New Done Done false 7302 Acked 8301 Flush 0 New Done Done false 7303 Acked 8302 Read 4 New Done Done false 7304 Acked 8303 Read 1 New Done Done false 7305 Acked 8304 Write 8 New Done Done false 7306 Acked 8305 Read 6 New Done Done false 7307 Acked 8306 Read 10 New Done Done false 7308 Acked 8307 Write 10 New Done Done false 7309 Acked 8308 Write 7 New Done Done false 7310 Acked 8309 Read 9 New Done Done false 7311 Acked 8310 Read 8 New Done Done false 7312 Acked 8311 Read 6 New Done Done false 7313 Acked 8312 Read 3 New Done Done false 7314 Acked 8313 Write 4 New Done Done false 7315 Acked 8314 Write 6 New Done Done false 7316 Acked 8315 Read 9 New Done Done false 7317 Acked 8316 Read 3 New Done Done false 7318 Acked 8317 Flush 0 New Done Done false 7319 Acked 8318 Write 6 New Done Done false 7320 Acked 8319 Write 5 New Done Done false 7321 Acked 8320 Read 4 New Done Done false 7322 Acked 8321 Read 9 New Done Done false 7323 Acked 8322 Write 8 New Done Done false 7324 Acked 8323 Write 4 New Done Done false 7325 Acked 8324 Read 10 New Done Done false 7326 Acked 8325 Read 3 New Done Done false 7327 Acked 8326 Read 8 New Done Done false 7328 Acked 8327 Flush 0 New Done Done false 7329 Acked 8328 Write 6 New Done Done false 7330 Acked 8329 Flush 0 New Done Done false 7331 Acked 8330 Read 1 New Done Done false 7332 Acked 8331 Read 5 New Done Done false 7333 Acked 8332 Read 7 New Done Done false 7334 Acked 8333 Write 3 New Done Done false 7335 Acked 8334 Read 7 New Done Done false 7336 Acked 8335 Write 10 New Done Done false 7337 Acked 8336 Read 3 New Done Done false 7338 Acked 8337 Write 3 New Done Done false 7339 Acked 8338 Flush 0 New Done Done false 7340 Acked 8339 Read 2 New Done Done false 7341 Acked 8340 Read 7 New Done Done false 7342 Acked 8341 Write 1 New Done Done false 7343 Acked 8342 Read 8 New Done Done false 7344 Acked 8343 Read 7 New Done Done false 7345 Acked 8344 Write 3 New Done Done false 7346 Acked 8345 Write 2 New Done Done false 7347 Acked 8346 Read 9 New Done Done false 7348 Acked 8347 Read 9 New Done Done false 7349 Acked 8348 Write 5 New Done Done false 7350 Acked 8349 Read 10 New Done Done false 7351 Acked 8350 Write 5 New Done Done false 7352 Acked 8351 Write 6 New Done Done false 7353 Acked 8352 Read 9 New Done Done false 7354 Acked 8353 Read 4 New Done Done false 7355 Acked 8354 Write 2 New Done Done false 7356 Acked 8355 Write 10 New Done Done false 7357 Acked 8356 Write 5 New Done Done false 7358 Acked 8357 Read 2 New Done Done false 7359 Acked 8358 Write 5 New Done Done false 7360 Acked 8359 Write 3 New Done Done false 7361 Acked 8360 Read 9 New Done Done false 7362 Acked 8361 Write 10 New Done Done false 7363 Acked 8362 Write 10 New Done Done false 7364 Acked 8363 Write 2 New Done Done false 7365 Acked 8364 Read 6 New Done Done false 7366 Acked 8365 Write 10 New Done Done false 7367 Acked 8366 Read 1 New Done Done false 7368 Acked 8367 Write 4 New Done Done false 7369 Acked 8368 Write 6 New Done Done false 7370 Acked 8369 Write 8 New Done Done false 7371 Acked 8370 Write 7 New Done Done false 7372 Acked 8371 Read 2 New Done Done false 7373 Acked 8372 Flush 0 New Done Done false 7374 Acked 8373 Write 10 New Done Done false 7375 Acked 8374 Flush 0 New Done Done false 7376 Acked 8375 Write 9 New Done Done false 7377 Acked 8376 Write 2 New Done Done false 7378 Acked 8377 Read 5 New Done Done false 7379 Acked 8378 Write 7 New Done Done false 7380 Acked 8379 Write 2 New Done Done false 7381 Acked 8380 Read 3 New Done Done false 7382 Acked 8381 Flush 0 New Done Done false 7383 Acked 8382 Write 6 New Done Done false 7384 Acked 8383 Write 3 New Done Done false 7385 Acked 8384 Flush 0 New Done Done false 7386 Acked 8385 Read 3 New Done Done false 7387 Acked 8386 Read 9 New Done Done false 7388 Acked 8387 Read 6 New Done Done false 7389 Acked 8388 Read 7 New Done Done false 7390 Acked 8389 Read 5 New Done Done false 7391 Acked 8390 Read 6 New Done Done false 7392 Acked 8391 Write 2 New Done Done false 7393 Acked 8392 Write 10 New Done Done false 7394 Acked 8393 Write 6 New Done Done false 7395 Acked 8394 Read 5 New Done Done false 7396 Acked 8395 Write 7 New Done Done false 7397 Acked 8396 Read 2 New Done Done false 7398 Acked 8397 Write 6 New Done Done false 7399 Acked 8398 Read 5 New Done Done false 7400 Acked 8399 Read 2 New Done Done false 7401 Acked 8400 Write 4 New Done Done false 7402 Acked 8401 Flush 0 New Done Done false 7403 Acked 8402 Read 8 New Done Done false 7404 Acked 8403 Write 10 New Done Done false 7405 Acked 8404 Read 8 New Done Done false 7406 Acked 8405 Read 10 New Done Done false 7407 Acked 8406 Read 8 New Done Done false 7408 Acked 8407 Write 10 New Done Done false 7409 Acked 8408 Write 3 New Done Done false 7410 Acked 8409 Write 3 New Done Done false 7411 Acked 8410 Read 5 New Done Done false 7412 Acked 8411 Read 7 New Done Done false 7413 Acked 8412 Read 8 New Done Done false 7414 Acked 8413 Read 1 New Done Done false 7415 Acked 8414 Read 1 New Done Done false 7416 Acked 8415 Read 2 New Done Done false 7417 Acked 8416 Flush 0 New Done Done false 7418 Acked 8417 Write 10 New Done Done false 7419 Acked 8418 Write 2 New Done Done false 7420 Acked 8419 Read 7 New Done Done false 7421 Acked 8420 Read 7 New Done Done false 7422 Acked 8421 Flush 0 New Done Done false 7423 Acked 8422 Read 3 New Done Done false 7424 Acked 8423 Write 3 New Done Done false 7425 Acked 8424 Flush 0 New Done Done false 7426 Acked 8425 Write 10 New Done Done false 7427 Acked 8426 Read 6 New Done Done false 7428 Acked 8427 Read 9 New Done Done false 7429 Acked 8428 Write 6 New Done Done false 7430 Acked 8429 Write 7 New Done Done false 7431 Acked 8430 Write 8 New Done Done false 7432 Acked 8431 Read 3 New Done Done false 7433 Acked 8432 Write 5 New Done Done false 7434 Acked 8433 Write 8 New Done Done false 7435 Acked 8434 Read 8 New Done Done false 7436 Acked 8435 Read 6 New Done Done false 7437 Acked 8436 Read 1 New Done Done false 7438 Acked 8437 Read 5 New Done Done false 7439 Acked 8438 Write 6 New Done Done false 7440 Acked 8439 Read 5 New Done Done false 7441 Acked 8440 Write 2 New Done Done false 7442 Acked 8441 Read 5 New Done Done false 7443 Acked 8442 Write 8 New Done Done false 7444 Acked 8443 Write 9 New Done Done false 7445 Acked 8444 Write 1 New Done Done false 7446 Acked 8445 Flush 0 New Done Done false 7447 Acked 8446 Write 9 New Done Done false 7448 Acked 8447 Read 2 New Done Done false 7449 Acked 8448 Write 10 New Done Done false 7450 Acked 8449 Read 10 New Done Done false 7451 Acked 8450 Read 6 New Done Done false 7452 Acked 8451 Read 9 New Done Done false 7453 Acked 8452 Write 1 New Done Done false 7454 Acked 8453 Read 8 New Done Done false 7455 Acked 8454 Read 5 New Done Done false 7456 Acked 8455 Read 9 New Done Done false 7457 Acked 8456 Write 9 New Done Done false 7458 Acked 8457 Read 1 New Done Done false 7459 Acked 8458 Read 5 New Done Done false 7460 Acked 8459 Flush 0 New Done Done false 7461 Acked 8460 Read 10 New Done Done false 7462 Acked 8461 Read 6 New Done Done false 7463 Acked 8462 Flush 0 New Done Done false 7464 Acked 8463 Write 7 New Done Done false 7465 Acked 8464 Write 2 New Done Done false 7466 Acked 8465 Read 3 New Done Done false 7467 Acked 8466 Read 3 New Done Done false 7468 Acked 8467 Read 2 New Done Done false 7469 Acked 8468 Read 10 New Done Done false 7470 Acked 8469 Read 2 New Done Done false 7471 Acked 8470 Write 2 New Done Done false 7472 Acked 8471 Read 1 New Done Done false 7473 Acked 8472 Write 9 New Done Done false 7474 Acked 8473 Flush 0 New Done Done false 7475 Acked 8474 Write 7 New Done Done false 7476 Acked 8475 Read 7 New Done Done false 7477 Acked 8476 Write 3 New Done Done false 7478 Acked 8477 Write 6 New Done Done false 7479 Acked 8478 Read 3 New Done Done false 7480 Acked 8479 Flush 0 New Done Done false 7481 Acked 8480 Read 1 New Done Done false 7482 Acked 8481 Read 1 New Done Done false 7483 Acked 8482 Write 9 New Done Done false 7484 Acked 8483 Read 3 New Done Done false 7485 Acked 8484 Write 5 New Done Done false 7486 Acked 8485 Read 10 New Done Done false 7487 Acked 8486 Flush 0 New Done Done false 7488 Acked 8487 Write 3 New Done Done false 7489 Acked 8488 Write 8 New Done Done false 7490 Acked 8489 Read 8 New Done Done false 7491 Acked 8490 Read 4 New Done Done false 7492 Acked 8491 Write 4 New Done Done false 7493 Acked 8492 Read 7 New Done Done false 7494 Acked 8493 Write 4 New Done Done false 7495 Acked 8494 Read 8 New Done Done false 7496 Acked 8495 Read 5 New Done Done false 7497 Acked 8496 Write 3 New Done Done false 7498 Acked 8497 Read 4 New Done Done false 7499 Acked 8498 Write 6 New Done Done false 7500 Acked 8499 Write 1 New Done Done false 7501 Acked 8500 Read 2 New Done Done false 7502 Acked 8501 Write 7 New Done Done false 7503 Acked 8502 Write 3 New Done Done false 7504 Acked 8503 Read 6 New Done Done false 7505 Acked 8504 Write 8 New Done Done false 7506 Acked 8505 Read 5 New Done Done false 7507 Acked 8506 Write 1 New Done Done false 7508 Acked 8507 Read 10 New Done Done false 7509 Acked 8508 Write 2 New Done Done false 7510 Acked 8509 Write 5 New Done Done false 7511 Acked 8510 Write 9 New Done Done false 7512 Acked 8511 Write 4 New Done Done false 7513 Acked 8512 Write 10 New Done Done false 7514 Acked 8513 Flush 0 New Done Done false 7515 Acked 8514 Read 1 New Done Done false 7516 Acked 8515 Write 2 New Done Done false 7517 Acked 8516 Write 4 New Done Done false 7518 Acked 8517 Write 5 New Done Done false 7519 Acked 8518 Read 4 New Done Done false 7520 Acked 8519 Read 2 New Done Done false 7521 Acked 8520 Read 7 New Done Done false 7522 Acked 8521 Read 7 New Done Done false 7523 Acked 8522 Read 3 New Done Done false 7524 Acked 8523 Write 6 New Done Done false 7525 Acked 8524 Read 2 New Done Done false 7526 Acked 8525 Write 9 New Done Done false 7527 Acked 8526 Write 10 New Done Done false 7528 Acked 8527 Read 1 New Done Done false 7529 Acked 8528 Read 2 New Done Done false 7530 Acked 8529 Flush 0 New Done Done false 7531 Acked 8530 Read 2 New Done Done false 7532 Acked 8531 Read 9 New Done Done false 7533 Acked 8532 Read 6 New Done Done false 7534 Acked 8533 Flush 0 New Done Done false 7535 Acked 8534 Read 1 New Done Done false 7536 Acked 8535 Write 4 New Done Done false 7537 Acked 8536 Flush 0 New Done Done false 7538 Acked 8537 Read 7 New Done Done false 7539 Acked 8538 Write 8 New Done Done false 7540 Acked 8539 Write 7 New Done Done false 7541 Acked 8540 Write 5 New Done Done false 7542 Acked 8541 Write 2 New Done Done false 7543 Acked 8542 Flush 0 New Done Done false 7544 Acked 8543 Write 9 New Done Done false 7545 Acked 8544 Read 10 New Done Done false 7546 Acked 8545 Read 9 New Done Done false 7547 Acked 8546 Read 2 New Done Done false 7548 Acked 8547 Flush 0 New Done Done false 7549 Acked 8548 Write 4 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8247 8547 8547 Downstairs last five completed: 8247 8246 8245 8244 8243 Upstairs last five completed: 7549 7548 7547 7546 7545 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:25.493459998Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:25.493578669Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:25.493641538Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:25.49368812Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:25.493708857Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:25.49373163Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:25.494031897Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:26.494869135Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:26.494938563Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:26.494966511Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:26.494990854Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:26.495017646Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:26.495038027Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:26.495060603Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:26.495274819Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.4964549Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.496689595Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.498751132Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.49883223Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.498860372Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 8247","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.49921768Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.499254925Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.499278642Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.49930115Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:27.499323552Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 8549 8548 8547 8546 8545 Upstairs last five completed: 7550 7549 7548 7547 7546 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 [8810] Got stop action so:false kr:false Jan 02 22:35:29.089 INFO request completed, latency_us: 518, response_code: 204, uri: /stop/cid/0, method: POST, req_id: be627ef3-37ce-4bb3-b835-d95b489ca8be, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:35:29.090 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5bf5a0a6-1777-4025-9ba7-37d7ccb8caac, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:29.0903807Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:29.09044684Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:29.090490351Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:29.090514086Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:29.090596709Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:29.090629817Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:30.091169753Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:30.09124582Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:30.091275265Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:30.091302142Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:30.0913226Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:30.091364813Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:30.091464601Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:31.092502573Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:31.092592817Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:31.092620904Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:31.092647579Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:31.092668044Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:31.092761524Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:31.092814059Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:32.094020089Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:32.094085872Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:32.09411359Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:32.094138393Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:32.09416491Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:32.094185447Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:32.094221072Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:32.094346733Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:35:33.092 INFO request completed, latency_us: 260, response_code: 200, uri: /state/cid/0, method: GET, req_id: b7d1b200-20c6-47b5-91ed-42352ced1d17, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 720 len 8192 002/300 Read block 2164 len 20480 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:33.095674258Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:33.095741027Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:33.095788166Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:33.095816298Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:33.095847303Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 2 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:33.095870802Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:33.095892984Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 1945 len 36864 004/300 Read block 3727 len 28672 005/300 Write block 3674 len 40960 data: 5 4 2 3 6 5 6 6 5 4 006/300 Read block 1345 len 16384 007/300 Write block 368 len 28672 data: 8 6 4 3 4 5 5 008/300 Read block 268 len 28672 009/300 Read block 4238 len 36864 010/300 Read block 259 len 24576 011/300 Write block 3605 len 40960 data: 6 8 7 8 8 8 9 6 6 6 012/300 Read block 2740 len 40960 013/300 Read block 4486 len 28672 014/300 Write block 1387 len 16384 data: 4 3 6 6 015/300 Flush 016/300 Write block 1895 len 24576 data: 5 5 5 4 4 4 017/300 Read block 3435 len 36864 018/300 Read block 2126 len 20480 019/300 Write block 325 len 8192 data: 6 5 020/300 Write block 3298 len 20480 data: 4 4 5 5 6 021/300 Write block 4117 len 8192 data: 2 4 022/300 Read block 492 len 32768 023/300 Read block 102 len 16384 024/300 Write block 3236 len 12288 data: 7 9 8 025/300 Write block 402 len 28672 data: 5 6 6 4 3 3 2 026/300 Write block 3330 len 36864 data: 4 5 8 7 7 7 7 6 5 027/300 Flush 028/300 Write block 2402 len 24576 data: 5 4 4 4 5 6 029/300 Read block 3979 len 20480 030/300 Read block 3879 len 36864 031/300 Read block 3379 len 28672 032/300 Read block 582 len 36864 033/300 Read block 4652 len 36864 034/300 Write block 1241 len 12288 data: 5 6 7 035/300 Write block 819 len 40960 data: 6 5 4 6 6 6 5 5 4 4 036/300 Flush 037/300 Read block 1862 len 8192 038/300 Flush 039/300 Write block 2118 len 36864 data: 5 6 5 7 7 7 5 4 5 040/300 Read block 507 len 28672 041/300 Write block 3044 len 12288 data: 5 5 6 042/300 Read block 2753 len 8192 043/300 Read block 2869 len 4096 044/300 Read block 3489 len 32768 045/300 Read block 4562 len 40960 046/300 Read block 2950 len 20480 047/300 Read block 170 len 16384 048/300 Write block 4437 len 4096 data: 7 049/300 Write block 4001 len 28672 data: 4 4 7 5 6 5 7 050/300 Read block 3329 len 36864 051/300 Write block 3209 len 28672 data: 3 3 3 5 6 6 8 052/300 Write block 4810 len 28672 data: 4 2 2 3 2 3 3 053/300 Read block 1452 len 12288 054/300 Write block 1562 len 16384 data: 4 2 2 2 055/300 Flush 056/300 Read block 4434 len 40960 057/300 Write block 4382 len 20480 data: 4 6 6 7 7 058/300 Write block 786 len 12288 data: 3 3 4 059/300 Write block 946 len 24576 data: 7 4 6 6 6 7 060/300 Write block 4533 len 28672 data: 4 6 7 7 5 6 6 061/300 Read block 3135 len 24576 062/300 Read block 2530 len 16384 063/300 Read block 3966 len 12288 064/300 Write block 898 len 16384 data: 7 6 6 3 065/300 Read block 3235 len 40960 066/300 Read block 2082 len 20480 067/300 Read block 4884 len 16384 068/300 Write block 4944 len 36864 data: 4 4 2 2 2 2 2 3 4 069/300 Read block 3408 len 20480 070/300 Write block 1837 len 32768 data: 4 5 5 5 5 4 5 4 071/300 Write block 4279 len 4096 data: 4 072/300 Read block 2888 len 4096 073/300 Read block 1413 len 4096 074/300 Read block 239 len 20480 075/300 Flush 076/300 Read block 3510 len 32768 077/300 Write block 2896 len 4096 data: 7 078/300 Read block 682 len 32768 079/300 Write block 3962 len 20480 data: 7 6 6 4 3 080/300 Write block 4328 len 32768 data: 7 6 5 4 4 6 6 7 081/300 Read block 2878 len 16384 082/300 Write block 2387 len 16384 data: 7 7 8 8 083/300 Write block 139 len 40960 data: 5 6 6 5 5 6 6 7 6 7 084/300 Write block 3874 len 40960 data: 4 3 3 5 5 5 5 8 9 10 085/300 Write block 4335 len 4096 data: 8 086/300 Write block 832 len 8192 data: 5 5 087/300 Read block 3897 len 8192 088/300 Read block 785 len 16384 089/300 Write block 2393 len 12288 data: 7 6 5 090/300 Read block 1192 len 28672 091/300 Write block 3738 len 4096 data: 5 092/300 Write block 3791 len 24576 data: 9 10 7 7 5 4 093/300 Write block 3970 len 8192 data: 2 2 094/300 Read block 4070 len 24576 095/300 Read block 3208 len 32768 096/300 Read block 3741 len 40960 097/300 Read block 939 len 24576 098/300 Read block 1115 len 28672 099/300 Write block 4290 len 12288 data: 5 4 4 100/300 Read block 1631 len 8192 101/300 Read block 3248 len 36864 102/300 Read block 1274 len 32768 103/300 Read block 504 len 36864 104/300 Read block 231 len 24576 105/300 Flush 106/300 Write block 2940 len 4096 data: 2 107/300 Read block 4473 len 24576 108/300 Write block 3249 len 28672 data: 8 7 7 7 6 7 6 109/300 Read block 371 len 24576 110/300 Read block 4264 len 40960 111/300 Flush 112/300 Read block 1574 len 8192 113/300 Write block 1902 len 24576 data: 6 5 5 4 4 4 114/300 Write block 1086 len 8192 data: 5 4 115/300 Read block 2752 len 12288 116/300 Read block 3646 len 20480 117/300 Write block 4237 len 20480 data: 6 5 4 5 6 118/300 Read block 4340 len 20480 119/300 Read block 977 len 24576 120/300 Write block 2429 len 36864 data: 4 4 4 3 4 6 6 5 4 121/300 Write block 2653 len 4096 data: 8 122/300 Write block 1490 len 24576 data: 4 5 4 4 5 8 123/300 Read block 3357 len 28672 124/300 Write block 233 len 20480 data: 6 6 5 5 5 125/300 Read block 4989 len 12288 126/300 Flush 127/300 Read block 4932 len 20480 128/300 Write block 3913 len 4096 data: 5 129/300 Read block 2575 len 16384 130/300 Write block 1848 len 40960 data: 4 3 3 3 2 3 3 3 3 2 131/300 Read block 4287 len 36864 132/300 Read block 3860 len 32768 133/300 Read block 4765 len 36864 134/300 Read block 4356 len 28672 135/300 Write block 4681 len 8192 data: 3 4 136/300 Read block 4074 len 4096 137/300 Read block 720 len 24576 138/300 Write block 3010 len 4096 data: 6 139/300 Read block 2264 len 8192 140/300 Read block 4039 len 28672 141/300 Flush 142/300 Read block 1079 len 4096 143/300 Read block 2460 len 40960 144/300 Flush 145/300 Write block 4411 len 12288 data: 3 3 4 146/300 Read block 3231 len 40960 147/300 Write block 1217 len 32768 data: 4 5 4 5 5 5 6 6 148/300 Read block 954 len 8192 149/300 Write block 2873 len 4096 data: 6 150/300 Read block 2640 len 16384 151/300 Read block 2396 len 40960 152/300 Write block 2120 len 12288 data: 6 8 8 153/300 Write block 4767 len 12288 data: 5 5 4 154/300 Write block 4929 len 32768 data: 10 8 11 9 9 8 10 9 155/300 Write block 2714 len 28672 data: 6 5 7 6 5 4 5 156/300 Read block 2436 len 12288 157/300 Write block 599 len 36864 data: 6 6 6 6 4 4 4 4 4 158/300 Write block 920 len 16384 data: 3 3 4 6 159/300 Read block 2330 len 8192 160/300 Read block 4323 len 36864 161/300 Read block 613 len 8192 162/300 Write block 4811 len 40960 data: 3 3 4 3 4 4 3 4 4 5 163/300 Read block 4803 len 32768 164/300 Read block 3655 len 28672 165/300 Read block 1925 len 28672 166/300 Write block 3500 len 28672 data: 4 3 4 4 5 5 4 167/300 Read block 1 len 40960 168/300 Write block 3129 len 36864 data: 5 4 3 5 6 6 5 6 6 169/300 Flush 170/300 Write block 1439 len 40960 data: 2 3 4 3 5 5 4 4 4 4 171/300 Write block 3299 len 12288 data: 5 6 6 172/300 Read block 3027 len 8192 173/300 Write block 2443 len 40960 data: 6 6 7 9 9 7 5 5 5 3 174/300 Write block 2345 len 32768 data: 4 4 4 3 5 6 4 4 175/300 Write block 233 len 24576 data: 7 7 6 6 6 5 176/300 Write block 4614 len 32768 data: 7 6 7 5 3 5 6 6 177/300 Read block 3734 len 20480 178/300 Read block 3822 len 20480 179/300 Read block 2636 len 4096 180/300 Read block 4331 len 36864 181/300 Read block 1769 len 20480 182/300 Read block 2458 len 32768 183/300 Write block 4276 len 28672 data: 4 3 4 5 4 4 4 184/300 Read block 4224 len 24576 185/300 Read block 3246 len 28672 186/300 Read block 879 len 28672 187/300 Read block 3483 len 16384 188/300 Read block 3308 len 4096 189/300 Read block 686 len 20480 190/300 Write block 4010 len 36864 data: 11 9 8 8 9 6 6 6 6 191/300 Write block 4979 len 16384 data: 6 5 5 5 192/300 Read block 4335 len 36864 193/300 Read block 2472 len 16384 194/300 Write block 2621 len 4096 data: 7 195/300 Read block 1388 len 24576 196/300 Read block 1537 len 36864 197/300 Read block 1080 len 8192 198/300 Write block 2302 len 12288 data: 6 5 6 199/300 Read block 3433 len 28672 200/300 Read block 1375 len 4096 201/300 Write block 1555 len 20480 data: 4 6 5 7 6 202/300 Read block 374 len 16384 203/300 Read block 3074 len 32768 204/300 Read block 4940 len 4096 205/300 Write block 2038 len 12288 data: 6 3 3 206/300 Read block 4062 len 28672 207/300 Read block 3560 len 12288 208/300 Flush 209/300 Flush 210/300 Write block 4883 len 16384 data: 7 6 9 8 211/300 Read block 4977 len 32768 212/300 Write block 1595 len 12288 data: 5 6 6 213/300 Write block 3760 len 40960 data: 6 7 7 8 8 10 11 10 10 8 214/300 Read block 4443 len 28672 215/300 Read block 470 len 16384 216/300 Write block 3006 len 8192 data: 4 4 217/300 Write block 140 len 40960 data: 7 7 6 6 7 7 8 7 8 7 218/300 Read block 281 len 24576 219/300 Write block 4601 len 36864 data: 5 5 8 6 7 8 7 8 9 220/300 Write block 4083 len 16384 data: 6 6 6 5 221/300 Read block 2509 len 36864 222/300 Flush 223/300 Read block 3765 len 16384 224/300 Write block 1425 len 16384 data: 5 4 4 5 225/300 Write block 415 len 20480 data: 4 6 6 8 8 226/300 Flush 227/300 Read block 3774 len 8192 228/300 Write block 2217 len 12288 data: 2 2 2 229/300 Read block 4288 len 4096 230/300 Read block 820 len 24576 231/300 Write block 1003 len 4096 data: 4 232/300 Read block 987 len 24576 233/300 Read block 2007 len 12288 234/300 Read block 327 len 20480 235/300 Read block 1039 len 24576 236/300 Read block 4711 len 20480 237/300 Read block 2705 len 20480 238/300 Write block 973 len 4096 data: 5 239/300 Read block 682 len 28672 240/300 Write block 3846 len 20480 data: 4 4 4 2 2 241/300 Read block 3308 len 20480 242/300 Write block 2864 len 32768 data: 5 6 6 7 7 6 5 6 243/300 Write block 2917 len 16384 data: 7 5 5 6 244/300 Write block 652 len 4096 data: 4 245/300 Read block 469 len 36864 246/300 Write block 2783 len 20480 data: 5 5 5 5 6 247/300 Read block 2101 len 16384 248/300 Read block 2659 len 24576 249/300 Write block 1970 len 12288 data: 6 7 7 250/300 Read block 4743 len 4096 251/300 Read block 1979 len 12288 252/300 Read block 4996 len 16384 253/300 Write block 553 len 8192 data: 4 3 254/300 Read block 3421 len 40960 255/300 Read block 884 len 12288 256/300 Read block 1232 len 36864 257/300 Read block 3603 len 4096 258/300 Write block 2797 len 20480 data: 4 5 8 8 7 259/300 Write block 3061 len 4096 data: 8 260/300 Write block 4097 len 24576 data: 5 5 5 7 7 8 261/300 Read block 4098 len 24576 262/300 Write block 2781 len 8192 data: 5 5 263/300 Write block 3211 len 20480 data: 4 6 7 7 9 264/300 Read block 3885 len 8192 265/300 Flush 266/300 Write block 127 len 32768 data: 5 5 7 7 7 6 7 7 267/300 Flush 268/300 Write block 3631 len 24576 data: 4 4 5 6 5 5 269/300 Read block 3692 len 28672 270/300 Read block 3808 len 4096 271/300 Write block 460 len 32768 data: 5 5 5 5 4 5 4 4 272/300 Read block 282 len 20480 273/300 Write block 1812 len 8192 data: 3 3 274/300 Write block 3567 len 4096 data: 4 275/300 Read block 2003 len 4096 276/300 Read block 183 len 4096 277/300 Read block 4453 len 32768 278/300 Write block 1226 len 32768 data: 5 5 6 7 6 6 6 5 279/300 Read block 616 len 8192 280/300 Read block 824 len 12288 281/300 Write block 4368 len 32768 data: 4 4 4 4 4 4 4 5 282/300 Write block 885 len 16384 data: 5 4 4 3 283/300 Read block 432 len 4096 284/300 Write block 4133 len 4096 data: 5 285/300 Write block 4476 len 16384 data: 4 5 4 4 286/300 Read block 3772 len 32768 287/300 Write block 1742 len 28672 data: 4 6 6 7 7 7 6 288/300 Write block 4890 len 24576 data: 5 5 5 4 4 4 289/300 Flush 290/300 Read block 4406 len 4096 291/300 Write block 3977 len 40960 data: 6 6 6 6 5 4 5 7 8 7 292/300 Flush 293/300 Flush 294/300 Write block 2384 len 8192 data: 3 6 295/300 Write block 1853 len 12288 data: 4 4 4 296/300 Read block 4772 len 24576 297/300 Write block 2549 len 40960 data: 5 5 5 5 5 5 5 4 4 4 298/300 Read block 155 len 4096 299/300 Write block 540 len 16384 data: 8 8 7 6 300/300 Read block 3238 len 28672 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:35:33.746 INFO request completed, latency_us: 293, response_code: 204, uri: /start/cid/0, method: POST, req_id: 5b0f94a2-aa4f-45ef-997f-53493530a297, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 7551 Acked 8550 Read 2 New Done Done true 7552 Acked 8551 Read 5 New Done Done true 7553 Acked 8552 Read 9 New Done Done false 7554 Acked 8553 Read 7 New Done Done false 7555 Acked 8554 Write 10 New Done Done false 7556 Acked 8555 Read 4 New Done Done false 7557 Acked 8556 Write 7 New Done Done false 7558 Acked 8557 Read 7 New Done Done false 7559 Acked 8558 Read 9 New Done Done false 7560 Acked 8559 Read 6 New Done Done false 7561 Acked 8560 Write 10 New Done Done false 7562 Acked 8561 Read 10 New Done Done false 7563 Acked 8562 Read 7 New Done Done false 7564 Acked 8563 Write 4 New Done Done false 7565 Acked 8564 Flush 0 New Done Done false 7566 Acked 8565 Write 6 New Done Done false 7567 Acked 8566 Read 9 New Done Done false 7568 Acked 8567 Read 5 New Done Done false 7569 Acked 8568 Write 2 New Done Done false 7570 Acked 8569 Write 5 New Done Done false 7571 Acked 8570 Write 2 New Done Done false 7572 Acked 8571 Read 8 New Done Done false 7573 Acked 8572 Read 4 New Done Done false 7574 Acked 8573 Write 3 New Done Done false 7575 Acked 8574 Write 7 New Done Done false 7576 Acked 8575 Write 9 New Done Done false 7577 Acked 8576 Flush 0 New Done Done false 7578 Acked 8577 Write 6 New Done Done false 7579 Acked 8578 Read 5 New Done Done false 7580 Acked 8579 Read 9 New Done Done false 7581 Acked 8580 Read 7 New Done Done false 7582 Acked 8581 Read 9 New Done Done false 7583 Acked 8582 Read 9 New Done Done false 7584 Acked 8583 Write 3 New Done Done false 7585 Acked 8584 Write 10 New Done Done false 7586 Acked 8585 Flush 0 New Done Done false 7587 Acked 8586 Read 2 New Done Done false 7588 Acked 8587 Flush 0 New Done Done false 7589 Acked 8588 Write 9 New Done Done false 7590 Acked 8589 Read 7 New Done Done false 7591 Acked 8590 Write 3 New Done Done false 7592 Acked 8591 Read 2 New Done Done false 7593 Acked 8592 Read 1 New Done Done false 7594 Acked 8593 Read 8 New Done Done false 7595 Acked 8594 Read 10 New Done Done false 7596 Acked 8595 Read 5 New Done Done false 7597 Acked 8596 Read 4 New Done Done false 7598 Acked 8597 Write 1 New Done Done false 7599 Acked 8598 Write 7 New Done Done false 7600 Acked 8599 Read 9 New Done Done false 7601 Acked 8600 Write 7 New Done Done false 7602 Acked 8601 Write 7 New Done Done false 7603 Acked 8602 Read 3 New Done Done false 7604 Acked 8603 Write 4 New Done Done false 7605 Acked 8604 Flush 0 New Done Done false 7606 Acked 8605 Read 10 New Done Done false 7607 Acked 8606 Write 5 New Done Done false 7608 Acked 8607 Write 3 New Done Done false 7609 Acked 8608 Write 6 New Done Done false 7610 Acked 8609 Write 7 New Done Done false 7611 Acked 8610 Read 6 New Done Done false 7612 Acked 8611 Read 4 New Done Done false 7613 Acked 8612 Read 3 New Done Done false 7614 Acked 8613 Write 4 New Done Done false 7615 Acked 8614 Read 10 New Done Done false 7616 Acked 8615 Read 5 New Done Done false 7617 Acked 8616 Read 4 New Done Done false 7618 Acked 8617 Write 9 New Done Done false 7619 Acked 8618 Read 5 New Done Done false 7620 Acked 8619 Write 8 New Done Done false 7621 Acked 8620 Write 1 New Done Done false 7622 Acked 8621 Read 1 New Done Done false 7623 Acked 8622 Read 1 New Done Done false 7624 Acked 8623 Read 5 New Done Done false 7625 Acked 8624 Flush 0 New Done Done false 7626 Acked 8625 Read 8 New Done Done false 7627 Acked 8626 Write 1 New Done Done false 7628 Acked 8627 Read 8 New Done Done false 7629 Acked 8628 Write 5 New Done Done false 7630 Acked 8629 Write 8 New Done Done false 7631 Acked 8630 Read 4 New Done Done false 7632 Acked 8631 Write 4 New Done Done false 7633 Acked 8632 Write 10 New Done Done false 7634 Acked 8633 Write 10 New Done Done false 7635 Acked 8634 Write 1 New Done Done false 7636 Acked 8635 Write 2 New Done Done false 7637 Acked 8636 Read 2 New Done Done false 7638 Acked 8637 Read 4 New Done Done false 7639 Acked 8638 Write 3 New Done Done false 7640 Acked 8639 Read 7 New Done Done false 7641 Acked 8640 Write 1 New Done Done false 7642 Acked 8641 Write 6 New Done Done false 7643 Acked 8642 Write 2 New Done Done false 7644 Acked 8643 Read 6 New Done Done false 7645 Acked 8644 Read 8 New Done Done false 7646 Acked 8645 Read 10 New Done Done false 7647 Acked 8646 Read 6 New Done Done false 7648 Acked 8647 Read 7 New Done Done false 7649 Acked 8648 Write 3 New Done Done false 7650 Acked 8649 Read 2 New Done Done false 7651 Acked 8650 Read 9 New Done Done false 7652 Acked 8651 Read 8 New Done Done false 7653 Acked 8652 Read 9 New Done Done false 7654 Acked 8653 Read 6 New Done Done false 7655 Acked 8654 Flush 0 New Done Done false 7656 Acked 8655 Write 1 New Done Done false 7657 Acked 8656 Read 6 New Done Done false 7658 Acked 8657 Write 7 New Done Done false 7659 Acked 8658 Read 6 New Done Done false 7660 Acked 8659 Read 10 New Done Done false 7661 Acked 8660 Flush 0 New Done Done false 7662 Acked 8661 Read 2 New Done Done false 7663 Acked 8662 Write 6 New Done Done false 7664 Acked 8663 Write 2 New Done Done false 7665 Acked 8664 Read 3 New Done Done false 7666 Acked 8665 Read 5 New Done Done false 7667 Acked 8666 Write 5 New Done Done false 7668 Acked 8667 Read 5 New Done Done false 7669 Acked 8668 Read 6 New Done Done false 7670 Acked 8669 Write 9 New Done Done false 7671 Acked 8670 Write 1 New Done Done false 7672 Acked 8671 Write 6 New Done Done false 7673 Acked 8672 Read 7 New Done Done false 7674 Acked 8673 Write 5 New Done Done false 7675 Acked 8674 Read 3 New Done Done false 7676 Acked 8675 Flush 0 New Done Done false 7677 Acked 8676 Read 5 New Done Done false 7678 Acked 8677 Write 1 New Done Done false 7679 Acked 8678 Read 4 New Done Done false 7680 Acked 8679 Write 10 New Done Done false 7681 Acked 8680 Read 9 New Done Done false 7682 Acked 8681 Read 8 New Done Done false 7683 Acked 8682 Read 9 New Done Done false 7684 Acked 8683 Read 7 New Done Done false 7685 Acked 8684 Write 2 New Done Done false 7686 Acked 8685 Read 1 New Done Done false 7687 Acked 8686 Read 6 New Done Done false 7688 Acked 8687 Write 1 New Done Done false 7689 Acked 8688 Read 2 New Done Done false 7690 Acked 8689 Read 7 New Done Done false 7691 Acked 8690 Flush 0 New Done Done false 7692 Acked 8691 Flush 0 New Done Done false 7693 Acked 8692 Read 1 New Done Done false 7694 Acked 8693 Read 10 New Done Done false 7695 Acked 8694 Flush 0 New Done Done false 7696 Acked 8695 Write 3 New Done Done false 7697 Acked 8696 Read 10 New Done Done false 7698 Acked 8697 Write 8 New Done Done false 7699 Acked 8698 Read 2 New Done Done false 7700 Acked 8699 Write 1 New Done Done false 7701 Acked 8700 Read 4 New Done Done false 7702 Acked 8701 Read 10 New Done Done false 7703 Acked 8702 Write 3 New Done Done false 7704 Acked 8703 Write 3 New Done Done false 7705 Acked 8704 Write 8 New Done Done false 7706 Acked 8705 Write 7 New Done Done false 7707 Acked 8706 Read 3 New Done Done false 7708 Acked 8707 Write 9 New Done Done false 7709 Acked 8708 Write 4 New Done Done false 7710 Acked 8709 Read 2 New Done Done false 7711 Acked 8710 Read 9 New Done Done false 7712 Acked 8711 Read 2 New Done Done false 7713 Acked 8712 Write 10 New Done Done false 7714 Acked 8713 Read 8 New Done Done false 7715 Acked 8714 Read 7 New Done Done false 7716 Acked 8715 Read 7 New Done Done false 7717 Acked 8716 Write 7 New Done Done false 7718 Acked 8717 Read 10 New Done Done false 7719 Acked 8718 Write 9 New Done Done false 7720 Acked 8719 Flush 0 New Done Done false 7721 Acked 8720 Write 10 New Done Done false 7722 Acked 8721 Write 3 New Done Done false 7723 Acked 8722 Read 2 New Done Done false 7724 Acked 8723 Write 10 New Done Done false 7725 Acked 8724 Write 8 New Done Done false 7726 Acked 8725 Write 6 New Done Done false 7727 Acked 8726 Write 8 New Done Done false 7728 Acked 8727 Read 5 New Done Done false 7729 Acked 8728 Read 5 New Done Done false 7730 Acked 8729 Read 1 New Done Done false 7731 Acked 8730 Read 9 New Done Done false 7732 Acked 8731 Read 5 New Done Done false 7733 Acked 8732 Read 8 New Done Done false 7734 Acked 8733 Write 7 New Done Done false 7735 Acked 8734 Read 6 New Done Done false 7736 Acked 8735 Read 7 New Done Done false 7737 Acked 8736 Read 7 New Done Done false 7738 Acked 8737 Read 4 New Done Done false 7739 Acked 8738 Read 1 New Done Done false 7740 Acked 8739 Read 5 New Done Done false 7741 Acked 8740 Write 9 New Done Done false 7742 Acked 8741 Write 4 New Done Done false 7743 Acked 8742 Read 9 New Done Done false 7744 Acked 8743 Read 4 New Done Done false 7745 Acked 8744 Write 1 New Done Done false 7746 Acked 8745 Read 6 New Done Done false 7747 Acked 8746 Read 9 New Done Done false 7748 Acked 8747 Read 2 New Done Done false 7749 Acked 8748 Write 3 New Done Done false 7750 Acked 8749 Read 7 New Done Done false 7751 Acked 8750 Read 1 New Done Done false 7752 Acked 8751 Write 5 New Done Done false 7753 Acked 8752 Read 4 New Done Done false 7754 Acked 8753 Read 8 New Done Done false 7755 Acked 8754 Read 1 New Done Done false 7756 Acked 8755 Write 3 New Done Done false 7757 Acked 8756 Read 7 New Done Done false 7758 Acked 8757 Read 3 New Done Done false 7759 Acked 8758 Flush 0 New Done Done false 7760 Acked 8759 Flush 0 New Done Done false 7761 Acked 8760 Write 4 New Done Done false 7762 Acked 8761 Read 8 New Done Done false 7763 Acked 8762 Write 3 New Done Done false 7764 Acked 8763 Write 10 New Done Done false 7765 Acked 8764 Read 7 New Done Done false 7766 Acked 8765 Read 4 New Done Done false 7767 Acked 8766 Write 2 New Done Done false 7768 Acked 8767 Write 10 New Done Done false 7769 Acked 8768 Read 6 New Done Done false 7770 Acked 8769 Write 9 New Done Done false 7771 Acked 8770 Write 4 New Done Done false 7772 Acked 8771 Read 9 New Done Done false 7773 Acked 8772 Flush 0 New Done Done false 7774 Acked 8773 Read 4 New Done Done false 7775 Acked 8774 Write 4 New Done Done false 7776 Acked 8775 Write 5 New Done Done false 7777 Acked 8776 Flush 0 New Done Done false 7778 Acked 8777 Read 2 New Done Done false 7779 Acked 8778 Write 3 New Done Done false 7780 Acked 8779 Read 1 New Done Done false 7781 Acked 8780 Read 6 New Done Done false 7782 Acked 8781 Write 1 New Done Done false 7783 Acked 8782 Read 6 New Done Done false 7784 Acked 8783 Read 3 New Done Done false 7785 Acked 8784 Read 5 New Done Done false 7786 Acked 8785 Read 6 New Done Done false 7787 Acked 8786 Read 5 New Done Done false 7788 Acked 8787 Read 5 New Done Done false 7789 Acked 8788 Write 1 New Done Done false 7790 Acked 8789 Read 7 New Done Done false 7791 Acked 8790 Write 5 New Done Done false 7792 Acked 8791 Read 5 New Done Done false 7793 Acked 8792 Write 8 New Done Done false 7794 Acked 8793 Write 4 New Done Done false 7795 Acked 8794 Write 1 New Done Done false 7796 Acked 8795 Read 9 New Done Done false 7797 Acked 8796 Write 5 New Done Done false 7798 Acked 8797 Read 4 New Done Done false 7799 Acked 8798 Read 6 New Done Done false 7800 Acked 8799 Write 3 New Done Done false 7801 Acked 8800 Read 1 New Done Done false 7802 Acked 8801 Read 3 New Done Done false 7803 Acked 8802 Read 4 New Done Done false 7804 Acked 8803 Write 2 New Done Done false 7805 Acked 8804 Read 10 New Done Done false 7806 Acked 8805 Read 3 New Done Done false 7807 Acked 8806 Read 9 New Done Done false 7808 Acked 8807 Read 1 New Done Done false 7809 Acked 8808 Write 5 New Done Done false 7810 Acked 8809 Write 1 New Done Done false 7811 Acked 8810 Write 6 New Done Done false 7812 Acked 8811 Read 6 New Done Done false 7813 Acked 8812 Write 2 New Done Done false 7814 Acked 8813 Write 5 New Done Done false 7815 Acked 8814 Read 2 New Done Done false 7816 Acked 8815 Flush 0 New Done Done false 7817 Acked 8816 Write 8 New Done Done false 7818 Acked 8817 Flush 0 New Done Done false 7819 Acked 8818 Write 6 New Done Done false 7820 Acked 8819 Read 7 New Done Done false 7821 Acked 8820 Read 1 New Done Done false 7822 Acked 8821 Write 8 New Done Done false 7823 Acked 8822 Read 5 New Done Done false 7824 Acked 8823 Write 2 New Done Done false 7825 Acked 8824 Write 1 New Done Done false 7826 Acked 8825 Read 1 New Done Done false 7827 Acked 8826 Read 1 New Done Done false 7828 Acked 8827 Read 8 New Done Done false 7829 Acked 8828 Write 8 New Done Done false 7830 Acked 8829 Read 2 New Done Done false 7831 Acked 8830 Read 3 New Done Done false 7832 Acked 8831 Write 8 New Done Done false 7833 Acked 8832 Write 4 New Done Done false 7834 Acked 8833 Read 1 New Done Done false 7835 Acked 8834 Write 1 New Done Done false 7836 Acked 8835 Write 4 New Done Done false 7837 Acked 8836 Read 8 New Done Done false 7838 Acked 8837 Write 7 New Done Done false 7839 Acked 8838 Write 6 New Done Done false 7840 Acked 8839 Flush 0 New Done Done false 7841 Acked 8840 Read 1 New Done Done false 7842 Acked 8841 Write 10 New Done Done false 7843 Acked 8842 Flush 0 New Done Done false 7844 Acked 8843 Flush 0 New Done Done false 7845 Acked 8844 Write 2 New Done Done false 7846 Acked 8845 Write 3 New Done Done false 7847 Acked 8846 Read 6 New Done Done false 7848 Acked 8847 Write 10 New Done Done false 7849 Acked 8848 Read 1 New Done Done false 7850 Acked 8849 Write 4 New Done Done false 7851 Acked 8850 Read 7 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8549 8843 8843 Downstairs last five completed: 8549 8548 8547 8546 8545 Upstairs last five completed: 7851 7850 7849 7848 7847 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:34.09697558Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:34.09704363Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:34.097072009Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:34.097099281Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:34.097120033Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:34.097208218Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:34.097453416Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:35.098307855Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:35.098374883Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:35.098414544Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:35.098442661Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:35.098462995Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:35.098506313Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:35.098763184Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:36.099685274Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:36.099751781Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:36.09977981Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:36.09980684Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:36.099827418Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:36.099930494Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:36.100177421Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:37.102037832Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:37.102103002Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:37.102141561Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:37.102166658Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:37.102192624Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:37.102212855Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:37.10223533Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:37.102478313Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 7551 Acked 8550 Read 2 New Done Done true 7552 Acked 8551 Read 5 New Done Done true 7553 Acked 8552 Read 9 New Done Done true 7554 Acked 8553 Read 7 New Done Done true 7555 Acked 8554 Write 10 New Done Done true 7556 Acked 8555 Read 4 New Done Done true 7557 Acked 8556 Write 7 New Done Done true 7558 Acked 8557 Read 7 New Done Done true 7559 Acked 8558 Read 9 New Done Done true 7560 Acked 8559 Read 6 New Done Done true 7561 Acked 8560 Write 10 New Done Done true 7562 Acked 8561 Read 10 New Done Done true 7563 Acked 8562 Read 7 New Done Done true 7564 Acked 8563 Write 4 New Done Done true 7565 Acked 8564 Flush 0 New Done Done true 7566 Acked 8565 Write 6 New Done Done true 7567 Acked 8566 Read 9 New Done Done true 7568 Acked 8567 Read 5 New Done Done true 7569 Acked 8568 Write 2 New Done Done true 7570 Acked 8569 Write 5 New Done Done true 7571 Acked 8570 Write 2 New Done Done true 7572 Acked 8571 Read 8 New Done Done true 7573 Acked 8572 Read 4 New Done Done true 7574 Acked 8573 Write 3 New Done Done true 7575 Acked 8574 Write 7 New Done Done true 7576 Acked 8575 Write 9 New Done Done true 7577 Acked 8576 Flush 0 New Done Done true 7578 Acked 8577 Write 6 New Done Done true 7579 Acked 8578 Read 5 New Done Done true 7580 Acked 8579 Read 9 New Done Done true 7581 Acked 8580 Read 7 New Done Done true 7582 Acked 8581 Read 9 New Done Done true 7583 Acked 8582 Read 9 New Done Done true 7584 Acked 8583 Write 3 New Done Done true 7585 Acked 8584 Write 10 New Done Done true 7586 Acked 8585 Flush 0 New Done Done true 7587 Acked 8586 Read 2 New Done Done true 7588 Acked 8587 Flush 0 New Done Done true 7589 Acked 8588 Write 9 New Done Done true 7590 Acked 8589 Read 7 New Done Done true 7591 Acked 8590 Write 3 New Done Done true 7592 Acked 8591 Read 2 New Done Done true 7593 Acked 8592 Read 1 New Done Done true 7594 Acked 8593 Read 8 New Done Done true 7595 Acked 8594 Read 10 New Done Done true 7596 Acked 8595 Read 5 New Done Done true 7597 Acked 8596 Read 4 New Done Done true 7598 Acked 8597 Write 1 New Done Done true 7599 Acked 8598 Write 7 New Done Done true 7600 Acked 8599 Read 9 New Done Done true 7601 Acked 8600 Write 7 New Done Done true 7602 Acked 8601 Write 7 New Done Done true 7603 Acked 8602 Read 3 New Done Done true 7604 Acked 8603 Write 4 New Done Done true 7605 Acked 8604 Flush 0 New Done Done true 7606 Acked 8605 Read 10 New Done Done true 7607 Acked 8606 Write 5 New Done Done true 7608 Acked 8607 Write 3 New Done Done true 7609 Acked 8608 Write 6 New Done Done true 7610 Acked 8609 Write 7 New Done Done true 7611 Acked 8610 Read 6 New Done Done true 7612 Acked 8611 Read 4 New Done Done true 7613 Acked 8612 Read 3 New Done Done true 7614 Acked 8613 Write 4 New Done Done true 7615 Acked 8614 Read 10 New Done Done true 7616 Acked 8615 Read 5 New Done Done true 7617 Acked 8616 Read 4 New Done Done true 7618 Acked 8617 Write 9 New Done Done true 7619 Acked 8618 Read 5 New Done Done true 7620 Acked 8619 Write 8 New Done Done true 7621 Acked 8620 Write 1 New Done Done true 7622 Acked 8621 Read 1 New Done Done true 7623 Acked 8622 Read 1 New Done Done true 7624 Acked 8623 Read 5 New Done Done true 7625 Acked 8624 Flush 0 New Done Done true 7626 Acked 8625 Read 8 New Done Done true 7627 Acked 8626 Write 1 New Done Done true 7628 Acked 8627 Read 8 New Done Done true 7629 Acked 8628 Write 5 New Done Done true 7630 Acked 8629 Write 8 New Done Done true 7631 Acked 8630 Read 4 New Done Done true 7632 Acked 8631 Write 4 New Done Done true 7633 Acked 8632 Write 10 New Done Done true 7634 Acked 8633 Write 10 New Done Done true 7635 Acked 8634 Write 1 New Done Done true 7636 Acked 8635 Write 2 New Done Done true 7637 Acked 8636 Read 2 New Done Done true 7638 Acked 8637 Read 4 New Done Done true 7639 Acked 8638 Write 3 New Done Done true 7640 Acked 8639 Read 7 New Done Done true 7641 Acked 8640 Write 1 New Done Done true 7642 Acked 8641 Write 6 New Done Done true 7643 Acked 8642 Write 2 New Done Done true 7644 Acked 8643 Read 6 New Done Done true 7645 Acked 8644 Read 8 New Done Done true 7646 Acked 8645 Read 10 New Done Done true 7647 Acked 8646 Read 6 New Done Done true 7648 Acked 8647 Read 7 New Done Done true 7649 Acked 8648 Write 3 New Done Done true 7650 Acked 8649 Read 2 New Done Done true 7651 Acked 8650 Read 9 New Done Done true 7652 Acked 8651 Read 8 New Done Done true 7653 Acked 8652 Read 9 New Done Done true 7654 Acked 8653 Read 6 New Done Done true 7655 Acked 8654 Flush 0 New Done Done true 7656 Acked 8655 Write 1 New Done Done true 7657 Acked 8656 Read 6 New Done Done true 7658 Acked 8657 Write 7 New Done Done true 7659 Acked 8658 Read 6 New Done Done true 7660 Acked 8659 Read 10 New Done Done true 7661 Acked 8660 Flush 0 New Done Done true 7662 Acked 8661 Read 2 New Done Done true 7663 Acked 8662 Write 6 New Done Done true 7664 Acked 8663 Write 2 New Done Done true 7665 Acked 8664 Read 3 New Done Done true 7666 Acked 8665 Read 5 New Done Done true 7667 Acked 8666 Write 5 New Done Done true 7668 Acked 8667 Read 5 New Done Done true 7669 Acked 8668 Read 6 New Done Done true 7670 Acked 8669 Write 9 New Done Done true 7671 Acked 8670 Write 1 New Done Done true 7672 Acked 8671 Write 6 New Done Done true 7673 Acked 8672 Read 7 New Done Done true 7674 Acked 8673 Write 5 New Done Done true 7675 Acked 8674 Read 3 New Done Done true 7676 Acked 8675 Flush 0 New Done Done true 7677 Acked 8676 Read 5 New Done Done true 7678 Acked 8677 Write 1 New Done Done true 7679 Acked 8678 Read 4 New Done Done true 7680 Acked 8679 Write 10 New Done Done true 7681 Acked 8680 Read 9 New Done Done true 7682 Acked 8681 Read 8 New Done Done true 7683 Acked 8682 Read 9 New Done Done true 7684 Acked 8683 Read 7 New Done Done true 7685 Acked 8684 Write 2 New Done Done true 7686 Acked 8685 Read 1 New Done Done true 7687 Acked 8686 Read 6 New Done Done true 7688 Acked 8687 Write 1 New Done Done true 7689 Acked 8688 Read 2 New Done Done true 7690 Acked 8689 Read 7 New Done Done true 7691 Acked 8690 Flush 0 New Done Done true 7692 Acked 8691 Flush 0 New Done Done true 7693 Acked 8692 Read 1 New Done Done true 7694 Acked 8693 Read 10 New Done Done true 7695 Acked 8694 Flush 0 New Done Done true 7696 Acked 8695 Write 3 New Done Done true 7697 Acked 8696 Read 10 New Done Done true 7698 Acked 8697 Write 8 New Done Done true 7699 Acked 8698 Read 2 New Done Done true 7700 Acked 8699 Write 1 New Done Done true 7701 Acked 8700 Read 4 New Done Done true 7702 Acked 8701 Read 10 New Done Done true 7703 Acked 8702 Write 3 New Done Done true 7704 Acked 8703 Write 3 New Done Done true 7705 Acked 8704 Write 8 New Done Done true 7706 Acked 8705 Write 7 New Done Done true 7707 Acked 8706 Read 3 New Done Done true 7708 Acked 8707 Write 9 New Done Done true 7709 Acked 8708 Write 4 New Done Done true 7710 Acked 8709 Read 2 New Done Done true 7711 Acked 8710 Read 9 New Done Done true 7712 Acked 8711 Read 2 New Done Done true 7713 Acked 8712 Write 10 New Done Done true 7714 Acked 8713 Read 8 New Done Done true 7715 Acked 8714 Read 7 New Done Done true 7716 Acked 8715 Read 7 New Done Done true 7717 Acked 8716 Write 7 New Done Done true 7718 Acked 8717 Read 10 New Done Done true 7719 Acked 8718 Write 9 New Done Done true 7720 Acked 8719 Flush 0 New Done Done true 7721 Acked 8720 Write 10 New Done Done true 7722 Acked 8721 Write 3 New Done Done true 7723 Acked 8722 Read 2 New Done Done true 7724 Acked 8723 Write 10 New Done Done true 7725 Acked 8724 Write 8 New Done Done true 7726 Acked 8725 Write 6 New Done Done true 7727 Acked 8726 Write 8 New Done Done true 7728 Acked 8727 Read 5 New Done Done true 7729 Acked 8728 Read 5 New Done Done true 7730 Acked 8729 Read 1 New Done Done true 7731 Acked 8730 Read 9 New Done Done true 7732 Acked 8731 Read 5 New Done Done true 7733 Acked 8732 Read 8 New Done Done true 7734 Acked 8733 Write 7 New Done Done true 7735 Acked 8734 Read 6 New Done Done true 7736 Acked 8735 Read 7 New Done Done true 7737 Acked 8736 Read 7 New Done Done true 7738 Acked 8737 Read 4 New Done Done true 7739 Acked 8738 Read 1 New Done Done true 7740 Acked 8739 Read 5 New Done Done true 7741 Acked 8740 Write 9 New Done Done true 7742 Acked 8741 Write 4 New Done Done true 7743 Acked 8742 Read 9 New Done Done true 7744 Acked 8743 Read 4 New Done Done true 7745 Acked 8744 Write 1 New Done Done true 7746 Acked 8745 Read 6 New Done Done true 7747 Acked 8746 Read 9 New Done Done true 7748 Acked 8747 Read 2 New Done Done true 7749 Acked 8748 Write 3 New Done Done true 7750 Acked 8749 Read 7 New Done Done true 7751 Acked 8750 Read 1 New Done Done true 7752 Acked 8751 Write 5 New Done Done true 7753 Acked 8752 Read 4 New Done Done true 7754 Acked 8753 Read 8 New Done Done true 7755 Acked 8754 Read 1 New Done Done true 7756 Acked 8755 Write 3 New Done Done true 7757 Acked 8756 Read 7 New Done Done true 7758 Acked 8757 Read 3 New Done Done true 7759 Acked 8758 Flush 0 New Done Done true 7760 Acked 8759 Flush 0 New Done Done true 7761 Acked 8760 Write 4 New Done Done true 7762 Acked 8761 Read 8 New Done Done true 7763 Acked 8762 Write 3 New Done Done true 7764 Acked 8763 Write 10 New Done Done true 7765 Acked 8764 Read 7 New Done Done true 7766 Acked 8765 Read 4 New Done Done true 7767 Acked 8766 Write 2 New Done Done true 7768 Acked 8767 Write 10 New Done Done true 7769 Acked 8768 Read 6 New Done Done true 7770 Acked 8769 Write 9 New Done Done true 7771 Acked 8770 Write 4 New Done Done true 7772 Acked 8771 Read 9 New Done Done true 7773 Acked 8772 Flush 0 New Done Done true 7774 Acked 8773 Read 4 New Done Done true 7775 Acked 8774 Write 4 New Done Done true 7776 Acked 8775 Write 5 New Done Done true 7777 Acked 8776 Flush 0 New Done Done true 7778 Acked 8777 Read 2 New Done Done true 7779 Acked 8778 Write 3 New Done Done true 7780 Acked 8779 Read 1 New Done Done true 7781 Acked 8780 Read 6 New Done Done true 7782 Acked 8781 Write 1 New Done Done true 7783 Acked 8782 Read 6 New Done Done true 7784 Acked 8783 Read 3 New Done Done true 7785 Acked 8784 Read 5 New Done Done true 7786 Acked 8785 Read 6 New Done Done true 7787 Acked 8786 Read 5 New Done Done true 7788 Acked 8787 Read 5 New Done Done true 7789 Acked 8788 Write 1 New Done Done true 7790 Acked 8789 Read 7 New Done Done true 7791 Acked 8790 Write 5 New Done Done true 7792 Acked 8791 Read 5 New Done Done true 7793 Acked 8792 Write 8 New Done Done true 7794 Acked 8793 Write 4 New Done Done true 7795 Acked 8794 Write 1 New Done Done true 7796 Acked 8795 Read 9 New Done Done true 7797 Acked 8796 Write 5 New Done Done true 7798 Acked 8797 Read 4 New Done Done true 7799 Acked 8798 Read 6 New Done Done true 7800 Acked 8799 Write 3 New Done Done true 7801 Acked 8800 Read 1 New Done Done true 7802 Acked 8801 Read 3 New Done Done true 7803 Acked 8802 Read 4 New Done Done true 7804 Acked 8803 Write 2 New Done Done true 7805 Acked 8804 Read 10 New Done Done true 7806 Acked 8805 Read 3 New Done Done true 7807 Acked 8806 Read 9 New Done Done true 7808 Acked 8807 Read 1 New Done Done true 7809 Acked 8808 Write 5 New Done Done true 7810 Acked 8809 Write 1 New Done Done true 7811 Acked 8810 Write 6 New Done Done true 7812 Acked 8811 Read 6 New Done Done true 7813 Acked 8812 Write 2 New Done Done true 7814 Acked 8813 Write 5 New Done Done true 7815 Acked 8814 Read 2 New Done Done true 7816 Acked 8815 Flush 0 New Done Done true 7817 Acked 8816 Write 8 New Done Done true 7818 Acked 8817 Flush 0 New Done Done true 7819 Acked 8818 Write 6 New Done Done true 7820 Acked 8819 Read 7 New Done Done true 7821 Acked 8820 Read 1 New Done Done true 7822 Acked 8821 Write 8 New Done Done true 7823 Acked 8822 Read 5 New Done Done true 7824 Acked 8823 Write 2 New Done Done true 7825 Acked 8824 Write 1 New Done Done true 7826 Acked 8825 Read 1 New Done Done true 7827 Acked 8826 Read 1 New Done Done true 7828 Acked 8827 Read 8 New Done Done true 7829 Acked 8828 Write 8 New Done Done true 7830 Acked 8829 Read 2 New Done Done true 7831 Acked 8830 Read 3 New Done Done true 7832 Acked 8831 Write 8 New Done Done true 7833 Acked 8832 Write 4 New Done Done true 7834 Acked 8833 Read 1 New Done Done true 7835 Acked 8834 Write 1 New Done Done true 7836 Acked 8835 Write 4 New Done Done true 7837 Acked 8836 Read 8 New Done Done true 7838 Acked 8837 Write 7 New Done Done true 7839 Acked 8838 Write 6 New Done Done true 7840 Acked 8839 Flush 0 New Done Done true 7841 Acked 8840 Read 1 New Done Done true 7842 Acked 8841 Write 10 New Done Done true 7843 Acked 8842 Flush 0 New Done Done true 7844 Acked 8843 Flush 0 New Done Done true 7845 Acked 8844 Write 2 New Done Done true 7846 Acked 8845 Write 3 New Done Done true 7847 Acked 8846 Read 6 New Done Done true 7848 Acked 8847 Write 10 New Done Done true 7849 Acked 8848 Read 1 New Done Done true 7850 Acked 8849 Write 4 New Done Done true 7851 Acked 8850 Read 7 New Done Done true 7852 Acked 8851 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 302 0 0 302 Sent 0 0 0 0 Done 0 302 302 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8549 8851 8851 Downstairs last five completed: 8549 8548 8547 8546 8545 Upstairs last five completed: 7852 7851 7850 7849 7848 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:38.103614395Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:38.103676213Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:38.1037035Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:38.103734703Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:38.103755904Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:38.103814485Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:38.104068072Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:39.104884971Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:39.104949947Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:39.10497804Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:39.105013525Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:39.105035223Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:39.105061487Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:39.105323767Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1211) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.10613367Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.106354512Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.10832825Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.108360482Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.108384173Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 8549","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.108678333Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.108707755Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.108731109Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.108753423Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:40.108775826Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 8851 8850 8849 8848 8847 Upstairs last five completed: 7852 7851 7850 7849 7848 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 [8810] Got stop action so:false kr:false Jan 02 22:35:41.754 INFO request completed, latency_us: 446, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 4f115a41-7636-4128-8b06-b1592091f616, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:41.755775988Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:35:41.755 {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:41.755873131Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} INFO request completed, latency_us: 190, response_code: 200, uri: /state/cid/0{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:41.755969297Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} , method: GET, req_id: 0178734f-7eed-4a4c-85a7-572f3f587ea9, remote_addr: 127.0.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:41.756051742Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 0.1:37434, local_addr: 127.0.0.1:9998{"msg":"[0] client re-new 0 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:41.756138721Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:41.756180914Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:42.757407992Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:42.757474865Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:42.75750345Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:42.75753023Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:42.757550716Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:42.75758391Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:42.757628704Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:43.758786038Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:43.7588564Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:43.758885168Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:43.758911924Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:43.758932476Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:43.75902613Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:43.759071687Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:44.760952419Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:44.761016404Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:44.761044801Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:44.761071548Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:44.761097276Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:44.761128336Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:44.761194114Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:35:45.757 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/0, method: GET, req_id: 82d660b3-5987-4522-8713-1c1b44c3a8dc, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 3996 len 20480 data: 11 9 7 4 4 002/300 Flush {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:45.762082967Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:45.762153088Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:45.76218991Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:45.762217685Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:45.762238759Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 2 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:45.762261434Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:45.762297068Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 4997 len 4096 004/300 Write block 288 len 8192 data: 8 8 005/300 Write block 1394 len 16384 data: 8 8 8 3 006/300 Write block 2461 len 20480 data: 6 7 7 7 7 007/300 Read block 130 len 32768 008/300 Flush 009/300 Write block 498 len 24576 data: 4 3 3 3 3 3 010/300 Read block 4188 len 8192 011/300 Read block 321 len 28672 012/300 Read block 3298 len 20480 013/300 Write block 1308 len 8192 data: 2 2 014/300 Read block 4810 len 36864 015/300 Read block 692 len 28672 016/300 Read block 2164 len 32768 017/300 Write block 2196 len 24576 data: 10 9 8 8 8 8 018/300 Write block 4537 len 28672 data: 6 7 7 6 6 6 4 019/300 Read block 558 len 16384 020/300 Write block 4264 len 8192 data: 6 8 021/300 Read block 1398 len 12288 022/300 Read block 3432 len 4096 023/300 Read block 910 len 32768 024/300 Write block 2943 len 24576 data: 4 5 6 6 7 10 025/300 Read block 3833 len 32768 026/300 Read block 2208 len 8192 027/300 Read block 2379 len 36864 028/300 Write block 2336 len 32768 data: 7 8 7 6 7 4 5 5 029/300 Write block 3097 len 28672 data: 4 6 7 6 6 7 6 030/300 Read block 3000 len 36864 031/300 Flush 032/300 Read block 4809 len 16384 033/300 Write block 53 len 28672 data: 5 5 4 4 4 4 4 034/300 Write block 4605 len 8192 data: 8 9 035/300 Read block 3505 len 36864 036/300 Write block 1647 len 4096 data: 4 037/300 Flush 038/300 Read block 4761 len 12288 039/300 Read block 1570 len 4096 040/300 Write block 3814 len 40960 data: 7 7 7 7 6 5 5 5 3 5 041/300 Write block 4085 len 40960 data: 7 6 6 6 6 8 9 9 7 8 042/300 Write block 1022 len 32768 data: 6 7 6 6 5 5 5 4 043/300 Read block 789 len 28672 044/300 Write block 3511 len 40960 data: 4 4 7 4 5 4 4 4 3 2 045/300 Read block 4448 len 16384 046/300 Write block 1475 len 24576 data: 3 5 4 4 5 4 047/300 Read block 3172 len 24576 048/300 Read block 4780 len 4096 049/300 Write block 1053 len 24576 data: 5 5 3 3 2 3 050/300 Read block 313 len 16384 051/300 Write block 285 len 16384 data: 6 8 8 9 052/300 Read block 1232 len 32768 053/300 Read block 589 len 20480 054/300 Write block 815 len 12288 data: 4 4 5 055/300 Read block 3436 len 28672 056/300 Read block 736 len 16384 057/300 Read block 4358 len 24576 058/300 Write block 710 len 20480 data: 8 9 8 8 7 059/300 Read block 3 len 32768 060/300 Read block 3227 len 12288 061/300 Flush 062/300 Write block 403 len 12288 data: 7 7 5 063/300 Write block 1560 len 28672 data: 5 6 5 3 3 3 2 064/300 Read block 457 len 4096 065/300 Read block 3124 len 20480 066/300 Read block 188 len 32768 067/300 Read block 4008 len 28672 068/300 Read block 4467 len 24576 069/300 Read block 3640 len 8192 070/300 Write block 576 len 24576 data: 6 7 7 7 7 5 071/300 Write block 1927 len 36864 data: 3 4 4 3 3 3 2 3 3 072/300 Read block 4952 len 24576 073/300 Read block 1562 len 8192 074/300 Read block 840 len 40960 075/300 Write block 4123 len 24576 data: 5 6 7 7 9 7 076/300 Write block 1220 len 12288 data: 6 6 6 077/300 Read block 2614 len 20480 078/300 Read block 695 len 20480 079/300 Read block 2845 len 4096 080/300 Read block 1201 len 40960 081/300 Read block 487 len 12288 082/300 Read block 1003 len 4096 083/300 Read block 47 len 28672 084/300 Read block 1996 len 40960 085/300 Read block 324 len 16384 086/300 Write block 1465 len 36864 data: 6 5 6 6 6 5 5 5 4 087/300 Write block 3729 len 32768 data: 7 7 7 4 4 4 4 5 088/300 Read block 1408 len 8192 089/300 Read block 3792 len 24576 090/300 Write block 1239 len 28672 data: 3 5 6 7 8 7 6 091/300 Write block 2130 len 28672 data: 5 4 4 4 4 2 3 092/300 Read block 3870 len 8192 093/300 Read block 4292 len 16384 094/300 Write block 1527 len 4096 data: 7 095/300 Read block 1343 len 24576 096/300 Write block 1061 len 12288 data: 3 3 4 097/300 Read block 4973 len 8192 098/300 Read block 2910 len 36864 099/300 Write block 2074 len 32768 data: 6 6 6 8 7 7 5 4 100/300 Read block 2429 len 40960 101/300 Write block 4486 len 28672 data: 4 4 4 5 4 4 5 102/300 Flush 103/300 Write block 4008 len 20480 data: 8 11 12 10 9 104/300 Read block 4150 len 28672 105/300 Write block 3602 len 4096 data: 7 106/300 Write block 1065 len 8192 data: 6 7 107/300 Read block 377 len 40960 108/300 Read block 2532 len 16384 109/300 Read block 3395 len 28672 110/300 Flush 111/300 Flush 112/300 Read block 3226 len 36864 113/300 Read block 3043 len 20480 114/300 Read block 754 len 28672 115/300 Write block 1615 len 16384 data: 3 3 4 3 116/300 Write block 363 len 8192 data: 7 9 117/300 Flush 118/300 Read block 2356 len 8192 119/300 Read block 3256 len 36864 120/300 Write block 4930 len 8192 data: 9 12 121/300 Read block 3336 len 40960 122/300 Read block 3527 len 24576 123/300 Read block 4364 len 20480 124/300 Write block 4432 len 4096 data: 7 125/300 Write block 2308 len 40960 data: 7 7 6 5 6 5 5 3 3 4 126/300 Read block 3611 len 32768 127/300 Write block 3865 len 36864 data: 5 5 3 5 5 5 5 4 4 128/300 Read block 912 len 8192 129/300 Flush 130/300 Flush 131/300 Read block 973 len 4096 132/300 Read block 2541 len 32768 133/300 Read block 3340 len 16384 134/300 Write block 407 len 16384 data: 4 3 4 4 135/300 Read block 2564 len 32768 136/300 Write block 434 len 8192 data: 7 7 137/300 Write block 1390 len 28672 data: 7 8 11 9 9 9 9 138/300 Read block 3175 len 16384 139/300 Read block 4180 len 24576 140/300 Write block 3295 len 8192 data: 3 3 141/300 Write block 4808 len 8192 data: 5 4 142/300 Flush 143/300 Read block 1397 len 28672 144/300 Flush 145/300 Write block 2142 len 36864 data: 5 6 6 6 5 6 7 6 6 146/300 Read block 2965 len 24576 147/300 Flush 148/300 Read block 1519 len 24576 149/300 Read block 177 len 4096 150/300 Read block 62 len 28672 151/300 Read block 3567 len 28672 152/300 Read block 4826 len 4096 153/300 Read block 3625 len 4096 154/300 Write block 1296 len 4096 data: 9 155/300 Read block 3883 len 20480 156/300 Write block 3189 len 12288 data: 3 5 5 157/300 Read block 1291 len 24576 158/300 Write block 2969 len 40960 data: 6 5 3 3 3 4 3 3 3 3 159/300 Read block 2258 len 20480 160/300 Read block 2180 len 12288 161/300 Read block 1897 len 12288 162/300 Read block 683 len 32768 163/300 Read block 2044 len 36864 164/300 Read block 4885 len 16384 165/300 Write block 1942 len 28672 data: 3 4 4 4 6 6 6 166/300 Read block 906 len 36864 167/300 Read block 2363 len 12288 168/300 Read block 2500 len 12288 169/300 Write block 2612 len 4096 data: 5 170/300 Write block 3842 len 16384 data: 5 5 5 4 171/300 Write block 1250 len 8192 data: 5 4 172/300 Flush 173/300 Write block 2802 len 36864 data: 7 7 5 5 6 5 5 4 4 174/300 Read block 1248 len 40960 175/300 Write block 2215 len 24576 data: 3 2 3 3 3 2 176/300 Read block 800 len 32768 177/300 Flush 178/300 Read block 2027 len 20480 179/300 Flush 180/300 Read block 2286 len 16384 181/300 Read block 172 len 40960 182/300 Read block 1528 len 24576 183/300 Flush 184/300 Read block 4378 len 40960 185/300 Read block 3006 len 40960 186/300 Read block 1060 len 20480 187/300 Read block 2198 len 28672 188/300 Flush 189/300 Read block 2701 len 8192 190/300 Write block 213 len 20480 data: 7 7 6 7 6 191/300 Write block 3158 len 12288 data: 3 3 3 192/300 Read block 610 len 8192 193/300 Write block 4252 len 16384 data: 8 8 8 8 194/300 Read block 82 len 24576 195/300 Write block 4536 len 24576 data: 8 7 8 8 7 7 196/300 Read block 3182 len 12288 197/300 Write block 1208 len 40960 data: 5 4 5 5 5 5 5 4 3 5 198/300 Write block 3056 len 28672 data: 8 8 8 9 10 9 8 199/300 Flush 200/300 Read block 3747 len 12288 201/300 Read block 1433 len 28672 202/300 Read block 1573 len 28672 203/300 Write block 4446 len 4096 data: 9 204/300 Write block 4018 len 32768 data: 7 5 5 4 3 4 4 5 205/300 Read block 1359 len 32768 206/300 Write block 1556 len 40960 data: 7 6 8 7 6 7 6 4 4 4 207/300 Write block 1627 len 36864 data: 8 5 7 7 8 6 6 6 8 208/300 Read block 3980 len 24576 209/300 Write block 1034 len 8192 data: 5 6 210/300 Write block 3707 len 28672 data: 4 4 5 4 5 4 4 211/300 Write block 2967 len 36864 data: 6 6 7 6 4 4 4 5 4 212/300 Flush 213/300 Read block 4402 len 16384 214/300 Read block 3 len 24576 215/300 Read block 3977 len 32768 216/300 Read block 4887 len 20480 217/300 Read block 4596 len 28672 218/300 Read block 2659 len 12288 219/300 Read block 2879 len 8192 220/300 Write block 205 len 16384 data: 6 5 6 7 221/300 Write block 442 len 24576 data: 7 6 5 6 4 3 222/300 Read block 2102 len 16384 223/300 Write block 1401 len 12288 data: 4 4 4 224/300 Read block 792 len 16384 225/300 Read block 4348 len 4096 226/300 Read block 776 len 12288 227/300 Read block 2425 len 32768 228/300 Read block 4381 len 40960 229/300 Write block 15 len 20480 data: 9 9 8 7 6 230/300 Read block 1040 len 20480 231/300 Read block 1365 len 40960 232/300 Read block 2614 len 28672 233/300 Write block 3330 len 24576 data: 5 6 9 8 8 8 234/300 Read block 488 len 36864 235/300 Read block 2558 len 36864 236/300 Flush 237/300 Read block 3094 len 8192 238/300 Flush 239/300 Read block 4499 len 16384 240/300 Write block 723 len 28672 data: 3 4 7 8 8 7 8 241/300 Read block 1690 len 32768 242/300 Write block 1255 len 32768 data: 9 10 10 10 9 8 10 8 243/300 Write block 2327 len 12288 data: 6 7 10 244/300 Read block 1777 len 28672 245/300 Write block 2998 len 16384 data: 3 3 3 3 246/300 Write block 601 len 40960 data: 7 7 5 5 5 5 5 4 4 4 247/300 Read block 459 len 24576 248/300 Write block 2219 len 36864 data: 4 3 4 3 4 4 5 7 6 249/300 Write block 2591 len 12288 data: 3 2 2 250/300 Read block 4339 len 16384 251/300 Flush 252/300 Read block 3225 len 36864 253/300 Flush 254/300 Read block 808 len 28672 255/300 Write block 3921 len 12288 data: 3 4 6 256/300 Read block 4204 len 8192 257/300 Write block 4490 len 32768 data: 5 5 6 5 5 4 4 4 258/300 Read block 1685 len 20480 259/300 Read block 3510 len 32768 260/300 Read block 572 len 24576 261/300 Read block 4828 len 20480 262/300 Write block 4404 len 4096 data: 3 263/300 Write block 4081 len 12288 data: 5 7 7 264/300 Flush 265/300 Read block 1517 len 40960 266/300 Read block 3030 len 36864 267/300 Read block 2952 len 24576 268/300 Read block 416 len 4096 269/300 Flush 270/300 Read block 4066 len 16384 271/300 Read block 4692 len 40960 272/300 Read block 1101 len 32768 273/300 Read block 3147 len 12288 274/300 Write block 4863 len 28672 data: 3 3 3 4 4 4 7 275/300 Write block 2017 len 16384 data: 6 5 4 5 276/300 Write block 3979 len 12288 data: 7 7 6 277/300 Flush 278/300 Read block 2605 len 8192 279/300 Read block 3985 len 28672 280/300 Write block 2964 len 40960 data: 5 6 5 7 7 8 7 5 5 5 281/300 Write block 4644 len 36864 data: 5 5 5 6 7 6 5 6 6 282/300 Write block 4308 len 16384 data: 6 6 6 4 283/300 Read block 10 len 12288 284/300 Write block 3673 len 12288 data: 4 6 5 285/300 Read block 1491 len 36864 286/300 Write block 2335 len 4096 data: 7 287/300 Write block 3877 len 16384 data: 6 6 6 6 288/300 Read block 1840 len 12288 289/300 Read block 2279 len 20480 290/300 Write block 3736 len 12288 data: 6 5 6 291/300 Read block 1160 len 20480 292/300 Write block 4622 len 8192 data: 6 6 293/300 Write block 4807 len 12288 data: 6 6 5 294/300 Read block 4346 len 40960 295/300 Flush 296/300 Write block 1434 len 8192 data: 3 3 297/300 Write block 529 len 40960 data: 7 6 6 7 6 8 11 11 10 9 298/300 Read block 2201 len 20480 299/300 Write block 457 len 16384 data: 4 3 5 6 300/300 Write block 3274 len 8192 data: 3 4 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:35:46.383 INFO request completed, latency_us: 324, response_code: 204, uri: /start/cid/0, method: POST, req_id: c0c1d050-cc86-4e3a-b338-ac21ab8258c8, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 7853 Acked 8852 Write 5 New Done Done true 7854 Acked 8853 Flush 0 New Done Done true 7855 Acked 8854 Read 1 New Done Done false 7856 Acked 8855 Write 2 New Done Done false 7857 Acked 8856 Write 4 New Done Done false 7858 Acked 8857 Write 5 New Done Done false 7859 Acked 8858 Read 8 New Done Done false 7860 Acked 8859 Flush 0 New Done Done false 7861 Acked 8860 Write 6 New Done Done false 7862 Acked 8861 Read 2 New Done Done false 7863 Acked 8862 Read 7 New Done Done false 7864 Acked 8863 Read 5 New Done Done false 7865 Acked 8864 Write 2 New Done Done false 7866 Acked 8865 Read 9 New Done Done false 7867 Acked 8866 Read 7 New Done Done false 7868 Acked 8867 Read 8 New Done Done false 7869 Acked 8868 Write 6 New Done Done false 7870 Acked 8869 Write 7 New Done Done false 7871 Acked 8870 Read 4 New Done Done false 7872 Acked 8871 Write 2 New Done Done false 7873 Acked 8872 Read 3 New Done Done false 7874 Acked 8873 Read 1 New Done Done false 7875 Acked 8874 Read 8 New Done Done false 7876 Acked 8875 Write 6 New Done Done false 7877 Acked 8876 Read 8 New Done Done false 7878 Acked 8877 Read 2 New Done Done false 7879 Acked 8878 Read 9 New Done Done false 7880 Acked 8879 Write 8 New Done Done false 7881 Acked 8880 Write 7 New Done Done false 7882 Acked 8881 Read 9 New Done Done false 7883 Acked 8882 Flush 0 New Done Done false 7884 Acked 8883 Read 4 New Done Done false 7885 Acked 8884 Write 7 New Done Done false 7886 Acked 8885 Write 2 New Done Done false 7887 Acked 8886 Read 9 New Done Done false 7888 Acked 8887 Write 1 New Done Done false 7889 Acked 8888 Flush 0 New Done Done false 7890 Acked 8889 Read 3 New Done Done false 7891 Acked 8890 Read 1 New Done Done false 7892 Acked 8891 Write 10 New Done Done false 7893 Acked 8892 Write 10 New Done Done false 7894 Acked 8893 Write 8 New Done Done false 7895 Acked 8894 Read 7 New Done Done false 7896 Acked 8895 Write 10 New Done Done false 7897 Acked 8896 Read 4 New Done Done false 7898 Acked 8897 Write 6 New Done Done false 7899 Acked 8898 Read 6 New Done Done false 7900 Acked 8899 Read 1 New Done Done false 7901 Acked 8900 Write 6 New Done Done false 7902 Acked 8901 Read 4 New Done Done false 7903 Acked 8902 Write 4 New Done Done false 7904 Acked 8903 Read 8 New Done Done false 7905 Acked 8904 Read 5 New Done Done false 7906 Acked 8905 Write 3 New Done Done false 7907 Acked 8906 Read 7 New Done Done false 7908 Acked 8907 Read 4 New Done Done false 7909 Acked 8908 Read 6 New Done Done false 7910 Acked 8909 Write 5 New Done Done false 7911 Acked 8910 Read 8 New Done Done false 7912 Acked 8911 Read 3 New Done Done false 7913 Acked 8912 Flush 0 New Done Done false 7914 Acked 8913 Write 3 New Done Done false 7915 Acked 8914 Write 7 New Done Done false 7916 Acked 8915 Read 1 New Done Done false 7917 Acked 8916 Read 5 New Done Done false 7918 Acked 8917 Read 8 New Done Done false 7919 Acked 8918 Read 7 New Done Done false 7920 Acked 8919 Read 6 New Done Done false 7921 Acked 8920 Read 2 New Done Done false 7922 Acked 8921 Write 6 New Done Done false 7923 Acked 8922 Write 9 New Done Done false 7924 Acked 8923 Read 6 New Done Done false 7925 Acked 8924 Read 2 New Done Done false 7926 Acked 8925 Read 10 New Done Done false 7927 Acked 8926 Write 6 New Done Done false 7928 Acked 8927 Write 3 New Done Done false 7929 Acked 8928 Read 5 New Done Done false 7930 Acked 8929 Read 5 New Done Done false 7931 Acked 8930 Read 1 New Done Done false 7932 Acked 8931 Read 10 New Done Done false 7933 Acked 8932 Read 3 New Done Done false 7934 Acked 8933 Read 1 New Done Done false 7935 Acked 8934 Read 7 New Done Done false 7936 Acked 8935 Read 10 New Done Done false 7937 Acked 8936 Read 4 New Done Done false 7938 Acked 8937 Write 9 New Done Done false 7939 Acked 8938 Write 8 New Done Done false 7940 Acked 8939 Read 2 New Done Done false 7941 Acked 8940 Read 6 New Done Done false 7942 Acked 8941 Write 7 New Done Done false 7943 Acked 8942 Write 7 New Done Done false 7944 Acked 8943 Read 2 New Done Done false 7945 Acked 8944 Read 4 New Done Done false 7946 Acked 8945 Write 1 New Done Done false 7947 Acked 8946 Read 6 New Done Done false 7948 Acked 8947 Write 3 New Done Done false 7949 Acked 8948 Read 2 New Done Done false 7950 Acked 8949 Read 9 New Done Done false 7951 Acked 8950 Write 8 New Done Done false 7952 Acked 8951 Read 10 New Done Done false 7953 Acked 8952 Write 7 New Done Done false 7954 Acked 8953 Flush 0 New Done Done false 7955 Acked 8954 Write 5 New Done Done false 7956 Acked 8955 Read 7 New Done Done false 7957 Acked 8956 Write 1 New Done Done false 7958 Acked 8957 Write 2 New Done Done false 7959 Acked 8958 Read 10 New Done Done false 7960 Acked 8959 Read 4 New Done Done false 7961 Acked 8960 Read 7 New Done Done false 7962 Acked 8961 Flush 0 New Done Done false 7963 Acked 8962 Flush 0 New Done Done false 7964 Acked 8963 Read 9 New Done Done false 7965 Acked 8964 Read 5 New Done Done false 7966 Acked 8965 Read 7 New Done Done false 7967 Acked 8966 Write 4 New Done Done false 7968 Acked 8967 Write 2 New Done Done false 7969 Acked 8968 Flush 0 New Done Done false 7970 Acked 8969 Read 2 New Done Done false 7971 Acked 8970 Read 9 New Done Done false 7972 Acked 8971 Write 2 New Done Done false 7973 Acked 8972 Read 10 New Done Done false 7974 Acked 8973 Read 6 New Done Done false 7975 Acked 8974 Read 5 New Done Done false 7976 Acked 8975 Write 1 New Done Done false 7977 Acked 8976 Write 10 New Done Done false 7978 Acked 8977 Read 8 New Done Done false 7979 Acked 8978 Write 9 New Done Done false 7980 Acked 8979 Read 2 New Done Done false 7981 Acked 8980 Flush 0 New Done Done false 7982 Acked 8981 Flush 0 New Done Done false 7983 Acked 8982 Read 1 New Done Done false 7984 Acked 8983 Read 8 New Done Done false 7985 Acked 8984 Read 4 New Done Done false 7986 Acked 8985 Write 4 New Done Done false 7987 Acked 8986 Read 8 New Done Done false 7988 Acked 8987 Write 2 New Done Done false 7989 Acked 8988 Write 7 New Done Done false 7990 Acked 8989 Read 4 New Done Done false 7991 Acked 8990 Read 6 New Done Done false 7992 Acked 8991 Write 2 New Done Done false 7993 Acked 8992 Write 2 New Done Done false 7994 Acked 8993 Flush 0 New Done Done false 7995 Acked 8994 Read 7 New Done Done false 7996 Acked 8995 Flush 0 New Done Done false 7997 Acked 8996 Write 9 New Done Done false 7998 Acked 8997 Read 6 New Done Done false 7999 Acked 8998 Flush 0 New Done Done false 8000 Acked 8999 Read 6 New Done Done false 8001 Acked 9000 Read 1 New Done Done false 8002 Acked 9001 Read 7 New Done Done false 8003 Acked 9002 Read 7 New Done Done false 8004 Acked 9003 Read 1 New Done Done false 8005 Acked 9004 Read 1 New Done Done false 8006 Acked 9005 Write 1 New Done Done false 8007 Acked 9006 Read 5 New Done Done false 8008 Acked 9007 Write 3 New Done Done false 8009 Acked 9008 Read 6 New Done Done false 8010 Acked 9009 Write 10 New Done Done false 8011 Acked 9010 Read 5 New Done Done false 8012 Acked 9011 Read 3 New Done Done false 8013 Acked 9012 Read 3 New Done Done false 8014 Acked 9013 Read 8 New Done Done false 8015 Acked 9014 Read 9 New Done Done false 8016 Acked 9015 Read 4 New Done Done false 8017 Acked 9016 Write 7 New Done Done false 8018 Acked 9017 Read 9 New Done Done false 8019 Acked 9018 Read 3 New Done Done false 8020 Acked 9019 Read 3 New Done Done false 8021 Acked 9020 Write 1 New Done Done false 8022 Acked 9021 Write 4 New Done Done false 8023 Acked 9022 Write 2 New Done Done false 8024 Acked 9023 Flush 0 New Done Done false 8025 Acked 9024 Write 9 New Done Done false 8026 Acked 9025 Read 10 New Done Done false 8027 Acked 9026 Write 6 New Done Done false 8028 Acked 9027 Read 8 New Done Done false 8029 Acked 9028 Flush 0 New Done Done false 8030 Acked 9029 Read 5 New Done Done false 8031 Acked 9030 Flush 0 New Done Done false 8032 Acked 9031 Read 4 New Done Done false 8033 Acked 9032 Read 10 New Done Done false 8034 Acked 9033 Read 6 New Done Done false 8035 Acked 9034 Flush 0 New Done Done false 8036 Acked 9035 Read 10 New Done Done false 8037 Acked 9036 Read 10 New Done Done false 8038 Acked 9037 Read 5 New Done Done false 8039 Acked 9038 Read 7 New Done Done false 8040 Acked 9039 Flush 0 New Done Done false 8041 Acked 9040 Read 2 New Done Done false 8042 Acked 9041 Write 5 New Done Done false 8043 Acked 9042 Write 3 New Done Done false 8044 Acked 9043 Read 2 New Done Done false 8045 Acked 9044 Write 4 New Done Done false 8046 Acked 9045 Read 6 New Done Done false 8047 Acked 9046 Write 6 New Done Done false 8048 Acked 9047 Read 3 New Done Done false 8049 Acked 9048 Write 10 New Done Done false 8050 Acked 9049 Write 7 New Done Done false 8051 Acked 9050 Flush 0 New Done Done false 8052 Acked 9051 Read 3 New Done Done false 8053 Acked 9052 Read 7 New Done Done false 8054 Acked 9053 Read 7 New Done Done false 8055 Acked 9054 Write 1 New Done Done false 8056 Acked 9055 Write 8 New Done Done false 8057 Acked 9056 Read 8 New Done Done false 8058 Acked 9057 Write 10 New Done Done false 8059 Acked 9058 Write 9 New Done Done false 8060 Acked 9059 Read 6 New Done Done false 8061 Acked 9060 Write 2 New Done Done false 8062 Acked 9061 Write 7 New Done Done false 8063 Acked 9062 Write 9 New Done Done false 8064 Acked 9063 Flush 0 New Done Done false 8065 Acked 9064 Read 4 New Done Done false 8066 Acked 9065 Read 6 New Done Done false 8067 Acked 9066 Read 8 New Done Done false 8068 Acked 9067 Read 5 New Done Done false 8069 Acked 9068 Read 7 New Done Done false 8070 Acked 9069 Read 3 New Done Done false 8071 Acked 9070 Read 2 New Done Done false 8072 Acked 9071 Write 4 New Done Done false 8073 Acked 9072 Write 6 New Done Done false 8074 Acked 9073 Read 4 New Done Done false 8075 Acked 9074 Write 3 New Done Done false 8076 Acked 9075 Read 4 New Done Done false 8077 Acked 9076 Read 1 New Done Done false 8078 Acked 9077 Read 3 New Done Done false 8079 Acked 9078 Read 8 New Done Done false 8080 Acked 9079 Read 10 New Done Done false 8081 Acked 9080 Write 5 New Done Done false 8082 Acked 9081 Read 5 New Done Done false 8083 Acked 9082 Read 10 New Done Done false 8084 Acked 9083 Read 7 New Done Done false 8085 Acked 9084 Write 6 New Done Done false 8086 Acked 9085 Read 9 New Done Done false 8087 Acked 9086 Read 9 New Done Done false 8088 Acked 9087 Flush 0 New Done Done false 8089 Acked 9088 Read 2 New Done Done false 8090 Acked 9089 Flush 0 New Done Done false 8091 Acked 9090 Read 4 New Done Done false 8092 Acked 9091 Write 7 New Done Done false 8093 Acked 9092 Read 8 New Done Done false 8094 Acked 9093 Write 8 New Done Done false 8095 Acked 9094 Write 3 New Done Done false 8096 Acked 9095 Read 7 New Done Done false 8097 Acked 9096 Write 4 New Done Done false 8098 Acked 9097 Write 10 New Done Done false 8099 Acked 9098 Read 6 New Done Done false 8100 Acked 9099 Write 9 New Done Done false 8101 Acked 9100 Write 3 New Done Done false 8102 Acked 9101 Read 4 New Done Done false 8103 Acked 9102 Flush 0 New Done Done false 8104 Acked 9103 Read 9 New Done Done false 8105 Acked 9104 Flush 0 New Done Done false 8106 Acked 9105 Read 7 New Done Done false 8107 Acked 9106 Write 3 New Done Done false 8108 Acked 9107 Read 2 New Done Done false 8109 Acked 9108 Write 8 New Done Done false 8110 Acked 9109 Read 5 New Done Done false 8111 Acked 9110 Read 8 New Done Done false 8112 Acked 9111 Read 6 New Done Done false 8113 Acked 9112 Read 5 New Done Done false 8114 Acked 9113 Write 1 New Done Done false 8115 Acked 9114 Write 3 New Done Done false 8116 Acked 9115 Flush 0 New Done Done false 8117 Acked 9116 Read 10 New Done Done false 8118 Acked 9117 Read 9 New Done Done false 8119 Acked 9118 Read 6 New Done Done false 8120 Acked 9119 Read 1 New Done Done false 8121 Acked 9120 Flush 0 New Done Done false 8122 Acked 9121 Read 4 New Done Done false 8123 Acked 9122 Read 10 New Done Done false 8124 Acked 9123 Read 8 New Done Done false 8125 Acked 9124 Read 3 New Done Done false 8126 Acked 9125 Write 7 New Done Done false 8127 Acked 9126 Write 4 New Done Done false 8128 Acked 9127 Write 3 New Done Done false 8129 Acked 9128 Flush 0 New Done Done false 8130 Acked 9129 Read 2 New Done Done false 8131 Acked 9130 Read 7 New Done Done false 8132 Acked 9131 Write 10 New Done Done false 8133 Acked 9132 Write 9 New Done Done false 8134 Acked 9133 Write 4 New Done Done false 8135 Acked 9134 Read 3 New Done Done false 8136 Acked 9135 Write 3 New Done Done false 8137 Acked 9136 Read 9 New Done Done false 8138 Acked 9137 Write 1 New Done Done false 8139 Acked 9138 Write 4 New Done Done false 8140 Acked 9139 Read 3 New Done Done false 8141 Acked 9140 Read 5 New Done Done false 8142 Acked 9141 Write 3 New Done Done false 8143 Acked 9142 Read 5 New Done Done false 8144 Acked 9143 Write 2 New Done Done false 8145 Acked 9144 Write 3 New Done Done false 8146 Acked 9145 Read 10 New Done Done false 8147 Acked 9146 Flush 0 New Done Done false 8148 Acked 9147 Write 2 New Done Done false 8149 Acked 9148 Write 10 New Done Done false 8150 Acked 9149 Read 5 New Done Done false 8151 Acked 9150 Write 4 New Done Done false 8152 Acked 9151 Write 2 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 300 0 0 300 Sent 0 0 0 0 Done 0 300 300 600 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8851 9146 9146 Downstairs last five completed: 8851 8850 8849 8848 8847 Upstairs last five completed: 8152 8151 8150 8149 8148 CLIENT: Up:0 ds:300 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:46.76377535Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:46.763848186Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:46.763878001Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:46.763912539Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:46.763939707Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:46.763960185Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:46.764000278Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:46.764278014Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:47.765471557Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:47.765567058Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:47.7655952Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:47.765622323Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:47.765642906Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:47.765673639Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:47.765945844Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:48.766897323Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:48.766962652Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:48.767000788Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:48.767028278Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:48.767048492Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:48.767079948Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:48.767334047Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:49.768178125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:49.768245508Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:49.768273698Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:49.768300503Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:49.768321266Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:49.768362484Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:49.768570258Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 7853 Acked 8852 Write 5 New Done Done true 7854 Acked 8853 Flush 0 New Done Done true 7855 Acked 8854 Read 1 New Done Done true 7856 Acked 8855 Write 2 New Done Done true 7857 Acked 8856 Write 4 New Done Done true 7858 Acked 8857 Write 5 New Done Done true 7859 Acked 8858 Read 8 New Done Done true 7860 Acked 8859 Flush 0 New Done Done true 7861 Acked 8860 Write 6 New Done Done true 7862 Acked 8861 Read 2 New Done Done true 7863 Acked 8862 Read 7 New Done Done true 7864 Acked 8863 Read 5 New Done Done true 7865 Acked 8864 Write 2 New Done Done true 7866 Acked 8865 Read 9 New Done Done true 7867 Acked 8866 Read 7 New Done Done true 7868 Acked 8867 Read 8 New Done Done true 7869 Acked 8868 Write 6 New Done Done true 7870 Acked 8869 Write 7 New Done Done true 7871 Acked 8870 Read 4 New Done Done true 7872 Acked 8871 Write 2 New Done Done true 7873 Acked 8872 Read 3 New Done Done true 7874 Acked 8873 Read 1 New Done Done true 7875 Acked 8874 Read 8 New Done Done true 7876 Acked 8875 Write 6 New Done Done true 7877 Acked 8876 Read 8 New Done Done true 7878 Acked 8877 Read 2 New Done Done true 7879 Acked 8878 Read 9 New Done Done true 7880 Acked 8879 Write 8 New Done Done true 7881 Acked 8880 Write 7 New Done Done true 7882 Acked 8881 Read 9 New Done Done true 7883 Acked 8882 Flush 0 New Done Done true 7884 Acked 8883 Read 4 New Done Done true 7885 Acked 8884 Write 7 New Done Done true 7886 Acked 8885 Write 2 New Done Done true 7887 Acked 8886 Read 9 New Done Done true 7888 Acked 8887 Write 1 New Done Done true 7889 Acked 8888 Flush 0 New Done Done true 7890 Acked 8889 Read 3 New Done Done true 7891 Acked 8890 Read 1 New Done Done true 7892 Acked 8891 Write 10 New Done Done true 7893 Acked 8892 Write 10 New Done Done true 7894 Acked 8893 Write 8 New Done Done true 7895 Acked 8894 Read 7 New Done Done true 7896 Acked 8895 Write 10 New Done Done true 7897 Acked 8896 Read 4 New Done Done true 7898 Acked 8897 Write 6 New Done Done true 7899 Acked 8898 Read 6 New Done Done true 7900 Acked 8899 Read 1 New Done Done true 7901 Acked 8900 Write 6 New Done Done true 7902 Acked 8901 Read 4 New Done Done true 7903 Acked 8902 Write 4 New Done Done true 7904 Acked 8903 Read 8 New Done Done true 7905 Acked 8904 Read 5 New Done Done true 7906 Acked 8905 Write 3 New Done Done true 7907 Acked 8906 Read 7 New Done Done true 7908 Acked 8907 Read 4 New Done Done true 7909 Acked 8908 Read 6 New Done Done true 7910 Acked 8909 Write 5 New Done Done true 7911 Acked 8910 Read 8 New Done Done true 7912 Acked 8911 Read 3 New Done Done true 7913 Acked 8912 Flush 0 New Done Done true 7914 Acked 8913 Write 3 New Done Done true 7915 Acked 8914 Write 7 New Done Done true 7916 Acked 8915 Read 1 New Done Done true 7917 Acked 8916 Read 5 New Done Done true 7918 Acked 8917 Read 8 New Done Done true 7919 Acked 8918 Read 7 New Done Done true 7920 Acked 8919 Read 6 New Done Done true 7921 Acked 8920 Read 2 New Done Done true 7922 Acked 8921 Write 6 New Done Done true 7923 Acked 8922 Write 9 New Done Done true 7924 Acked 8923 Read 6 New Done Done true 7925 Acked 8924 Read 2 New Done Done true 7926 Acked 8925 Read 10 New Done Done true 7927 Acked 8926 Write 6 New Done Done true 7928 Acked 8927 Write 3 New Done Done true 7929 Acked 8928 Read 5 New Done Done true 7930 Acked 8929 Read 5 New Done Done true 7931 Acked 8930 Read 1 New Done Done true 7932 Acked 8931 Read 10 New Done Done true 7933 Acked 8932 Read 3 New Done Done true 7934 Acked 8933 Read 1 New Done Done true 7935 Acked 8934 Read 7 New Done Done true 7936 Acked 8935 Read 10 New Done Done true 7937 Acked 8936 Read 4 New Done Done true 7938 Acked 8937 Write 9 New Done Done true 7939 Acked 8938 Write 8 New Done Done true 7940 Acked 8939 Read 2 New Done Done true 7941 Acked 8940 Read 6 New Done Done true 7942 Acked 8941 Write 7 New Done Done true 7943 Acked 8942 Write 7 New Done Done true 7944 Acked 8943 Read 2 New Done Done true 7945 Acked 8944 Read 4 New Done Done true 7946 Acked 8945 Write 1 New Done Done true 7947 Acked 8946 Read 6 New Done Done true 7948 Acked 8947 Write 3 New Done Done true 7949 Acked 8948 Read 2 New Done Done true 7950 Acked 8949 Read 9 New Done Done true 7951 Acked 8950 Write 8 New Done Done true 7952 Acked 8951 Read 10 New Done Done true 7953 Acked 8952 Write 7 New Done Done true 7954 Acked 8953 Flush 0 New Done Done true 7955 Acked 8954 Write 5 New Done Done true 7956 Acked 8955 Read 7 New Done Done true 7957 Acked 8956 Write 1 New Done Done true 7958 Acked 8957 Write 2 New Done Done true 7959 Acked 8958 Read 10 New Done Done true 7960 Acked 8959 Read 4 New Done Done true 7961 Acked 8960 Read 7 New Done Done true 7962 Acked 8961 Flush 0 New Done Done true 7963 Acked 8962 Flush 0 New Done Done true 7964 Acked 8963 Read 9 New Done Done true 7965 Acked 8964 Read 5 New Done Done true 7966 Acked 8965 Read 7 New Done Done true 7967 Acked 8966 Write 4 New Done Done true 7968 Acked 8967 Write 2 New Done Done true 7969 Acked 8968 Flush 0 New Done Done true 7970 Acked 8969 Read 2 New Done Done true 7971 Acked 8970 Read 9 New Done Done true 7972 Acked 8971 Write 2 New Done Done true 7973 Acked 8972 Read 10 New Done Done true 7974 Acked 8973 Read 6 New Done Done true 7975 Acked 8974 Read 5 New Done Done true 7976 Acked 8975 Write 1 New Done Done true 7977 Acked 8976 Write 10 New Done Done true 7978 Acked 8977 Read 8 New Done Done true 7979 Acked 8978 Write 9 New Done Done true 7980 Acked 8979 Read 2 New Done Done true 7981 Acked 8980 Flush 0 New Done Done true 7982 Acked 8981 Flush 0 New Done Done true 7983 Acked 8982 Read 1 New Done Done true 7984 Acked 8983 Read 8 New Done Done true 7985 Acked 8984 Read 4 New Done Done true 7986 Acked 8985 Write 4 New Done Done true 7987 Acked 8986 Read 8 New Done Done true 7988 Acked 8987 Write 2 New Done Done true 7989 Acked 8988 Write 7 New Done Done true 7990 Acked 8989 Read 4 New Done Done true 7991 Acked 8990 Read 6 New Done Done true 7992 Acked 8991 Write 2 New Done Done true 7993 Acked 8992 Write 2 New Done Done true 7994 Acked 8993 Flush 0 New Done Done true 7995 Acked 8994 Read 7 New Done Done true 7996 Acked 8995 Flush 0 New Done Done true 7997 Acked 8996 Write 9 New Done Done true 7998 Acked 8997 Read 6 New Done Done true 7999 Acked 8998 Flush 0 New Done Done true 8000 Acked 8999 Read 6 New Done Done true 8001 Acked 9000 Read 1 New Done Done true 8002 Acked 9001 Read 7 New Done Done true 8003 Acked 9002 Read 7 New Done Done true 8004 Acked 9003 Read 1 New Done Done true 8005 Acked 9004 Read 1 New Done Done true 8006 Acked 9005 Write 1 New Done Done true 8007 Acked 9006 Read 5 New Done Done true 8008 Acked 9007 Write 3 New Done Done true 8009 Acked 9008 Read 6 New Done Done true 8010 Acked 9009 Write 10 New Done Done true 8011 Acked 9010 Read 5 New Done Done true 8012 Acked 9011 Read 3 New Done Done true 8013 Acked 9012 Read 3 New Done Done true 8014 Acked 9013 Read 8 New Done Done true 8015 Acked 9014 Read 9 New Done Done true 8016 Acked 9015 Read 4 New Done Done true 8017 Acked 9016 Write 7 New Done Done true 8018 Acked 9017 Read 9 New Done Done true 8019 Acked 9018 Read 3 New Done Done true 8020 Acked 9019 Read 3 New Done Done true 8021 Acked 9020 Write 1 New Done Done true 8022 Acked 9021 Write 4 New Done Done true 8023 Acked 9022 Write 2 New Done Done true 8024 Acked 9023 Flush 0 New Done Done true 8025 Acked 9024 Write 9 New Done Done true 8026 Acked 9025 Read 10 New Done Done true 8027 Acked 9026 Write 6 New Done Done true 8028 Acked 9027 Read 8 New Done Done true 8029 Acked 9028 Flush 0 New Done Done true 8030 Acked 9029 Read 5 New Done Done true 8031 Acked 9030 Flush 0 New Done Done true 8032 Acked 9031 Read 4 New Done Done true 8033 Acked 9032 Read 10 New Done Done true 8034 Acked 9033 Read 6 New Done Done true 8035 Acked 9034 Flush 0 New Done Done true 8036 Acked 9035 Read 10 New Done Done true 8037 Acked 9036 Read 10 New Done Done true 8038 Acked 9037 Read 5 New Done Done true 8039 Acked 9038 Read 7 New Done Done true 8040 Acked 9039 Flush 0 New Done Done true 8041 Acked 9040 Read 2 New Done Done true 8042 Acked 9041 Write 5 New Done Done true 8043 Acked 9042 Write 3 New Done Done true 8044 Acked 9043 Read 2 New Done Done true 8045 Acked 9044 Write 4 New Done Done true 8046 Acked 9045 Read 6 New Done Done true 8047 Acked 9046 Write 6 New Done Done true 8048 Acked 9047 Read 3 New Done Done true 8049 Acked 9048 Write 10 New Done Done true 8050 Acked 9049 Write 7 New Done Done true 8051 Acked 9050 Flush 0 New Done Done true 8052 Acked 9051 Read 3 New Done Done true 8053 Acked 9052 Read 7 New Done Done true 8054 Acked 9053 Read 7 New Done Done true 8055 Acked 9054 Write 1 New Done Done true 8056 Acked 9055 Write 8 New Done Done true 8057 Acked 9056 Read 8 New Done Done true 8058 Acked 9057 Write 10 New Done Done true 8059 Acked 9058 Write 9 New Done Done true 8060 Acked 9059 Read 6 New Done Done true 8061 Acked 9060 Write 2 New Done Done true 8062 Acked 9061 Write 7 New Done Done true 8063 Acked 9062 Write 9 New Done Done true 8064 Acked 9063 Flush 0 New Done Done true 8065 Acked 9064 Read 4 New Done Done true 8066 Acked 9065 Read 6 New Done Done true 8067 Acked 9066 Read 8 New Done Done true 8068 Acked 9067 Read 5 New Done Done true 8069 Acked 9068 Read 7 New Done Done true 8070 Acked 9069 Read 3 New Done Done true 8071 Acked 9070 Read 2 New Done Done true 8072 Acked 9071 Write 4 New Done Done true 8073 Acked 9072 Write 6 New Done Done true 8074 Acked 9073 Read 4 New Done Done true 8075 Acked 9074 Write 3 New Done Done true 8076 Acked 9075 Read 4 New Done Done true 8077 Acked 9076 Read 1 New Done Done true 8078 Acked 9077 Read 3 New Done Done true 8079 Acked 9078 Read 8 New Done Done true 8080 Acked 9079 Read 10 New Done Done true 8081 Acked 9080 Write 5 New Done Done true 8082 Acked 9081 Read 5 New Done Done true 8083 Acked 9082 Read 10 New Done Done true 8084 Acked 9083 Read 7 New Done Done true 8085 Acked 9084 Write 6 New Done Done true 8086 Acked 9085 Read 9 New Done Done true 8087 Acked 9086 Read 9 New Done Done true 8088 Acked 9087 Flush 0 New Done Done true 8089 Acked 9088 Read 2 New Done Done true 8090 Acked 9089 Flush 0 New Done Done true 8091 Acked 9090 Read 4 New Done Done true 8092 Acked 9091 Write 7 New Done Done true 8093 Acked 9092 Read 8 New Done Done true 8094 Acked 9093 Write 8 New Done Done true 8095 Acked 9094 Write 3 New Done Done true 8096 Acked 9095 Read 7 New Done Done true 8097 Acked 9096 Write 4 New Done Done true 8098 Acked 9097 Write 10 New Done Done true 8099 Acked 9098 Read 6 New Done Done true 8100 Acked 9099 Write 9 New Done Done true 8101 Acked 9100 Write 3 New Done Done true 8102 Acked 9101 Read 4 New Done Done true 8103 Acked 9102 Flush 0 New Done Done true 8104 Acked 9103 Read 9 New Done Done true 8105 Acked 9104 Flush 0 New Done Done true 8106 Acked 9105 Read 7 New Done Done true 8107 Acked 9106 Write 3 New Done Done true 8108 Acked 9107 Read 2 New Done Done true 8109 Acked 9108 Write 8 New Done Done true 8110 Acked 9109 Read 5 New Done Done true 8111 Acked 9110 Read 8 New Done Done true 8112 Acked 9111 Read 6 New Done Done true 8113 Acked 9112 Read 5 New Done Done true 8114 Acked 9113 Write 1 New Done Done true 8115 Acked 9114 Write 3 New Done Done true 8116 Acked 9115 Flush 0 New Done Done true 8117 Acked 9116 Read 10 New Done Done true 8118 Acked 9117 Read 9 New Done Done true 8119 Acked 9118 Read 6 New Done Done true 8120 Acked 9119 Read 1 New Done Done true 8121 Acked 9120 Flush 0 New Done Done true 8122 Acked 9121 Read 4 New Done Done true 8123 Acked 9122 Read 10 New Done Done true 8124 Acked 9123 Read 8 New Done Done true 8125 Acked 9124 Read 3 New Done Done true 8126 Acked 9125 Write 7 New Done Done true 8127 Acked 9126 Write 4 New Done Done true 8128 Acked 9127 Write 3 New Done Done true 8129 Acked 9128 Flush 0 New Done Done true 8130 Acked 9129 Read 2 New Done Done true 8131 Acked 9130 Read 7 New Done Done true 8132 Acked 9131 Write 10 New Done Done true 8133 Acked 9132 Write 9 New Done Done true 8134 Acked 9133 Write 4 New Done Done true 8135 Acked 9134 Read 3 New Done Done true 8136 Acked 9135 Write 3 New Done Done true 8137 Acked 9136 Read 9 New Done Done true 8138 Acked 9137 Write 1 New Done Done true 8139 Acked 9138 Write 4 New Done Done true 8140 Acked 9139 Read 3 New Done Done true 8141 Acked 9140 Read 5 New Done Done true 8142 Acked 9141 Write 3 New Done Done true 8143 Acked 9142 Read 5 New Done Done true 8144 Acked 9143 Write 2 New Done Done true 8145 Acked 9144 Write 3 New Done Done true 8146 Acked 9145 Read 10 New Done Done true 8147 Acked 9146 Flush 0 New Done Done true 8148 Acked 9147 Write 2 New Done Done true 8149 Acked 9148 Write 10 New Done Done true 8150 Acked 9149 Read 5 New Done Done true 8151 Acked 9150 Write 4 New Done Done true 8152 Acked 9151 Write 2 New Done Done true 8153 Acked 9152 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8851 9152 9152 Downstairs last five completed: 8851 8850 8849 8848 8847 Upstairs last five completed: 8153 8152 8151 8150 8149 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:50.769887682Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:50.769953459Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:50.769981926Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:50.770008652Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:50.77003633Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:50.770064729Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:50.770308583Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:51.772040576Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:51.772108515Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:51.772136658Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:51.772160626Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:51.772186966Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:51.772207311Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:51.772237877Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:51.772485179Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:52.773488368Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:52.773554774Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:52.773582761Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:52.773619019Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:52.773640368Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:52.773662647Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:52.773917519Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1212) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:53.774859042Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:53.774925517Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:53.774953865Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:53.774980393Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:53.775001067Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 301 jobs since flush 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:53.775121229Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:53.775293434Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 7853 Acked 8852 Write 5 New Done Done true 7854 Acked 8853 Flush 0 New Done Done true 7855 Acked 8854 Read 1 New Done Done true 7856 Acked 8855 Write 2 New Done Done true 7857 Acked 8856 Write 4 New Done Done true 7858 Acked 8857 Write 5 New Done Done true 7859 Acked 8858 Read 8 New Done Done true 7860 Acked 8859 Flush 0 New Done Done true 7861 Acked 8860 Write 6 New Done Done true 7862 Acked 8861 Read 2 New Done Done true 7863 Acked 8862 Read 7 New Done Done true 7864 Acked 8863 Read 5 New Done Done true 7865 Acked 8864 Write 2 New Done Done true 7866 Acked 8865 Read 9 New Done Done true 7867 Acked 8866 Read 7 New Done Done true 7868 Acked 8867 Read 8 New Done Done true 7869 Acked 8868 Write 6 New Done Done true 7870 Acked 8869 Write 7 New Done Done true 7871 Acked 8870 Read 4 New Done Done true 7872 Acked 8871 Write 2 New Done Done true 7873 Acked 8872 Read 3 New Done Done true 7874 Acked 8873 Read 1 New Done Done true 7875 Acked 8874 Read 8 New Done Done true 7876 Acked 8875 Write 6 New Done Done true 7877 Acked 8876 Read 8 New Done Done true 7878 Acked 8877 Read 2 New Done Done true 7879 Acked 8878 Read 9 New Done Done true 7880 Acked 8879 Write 8 New Done Done true 7881 Acked 8880 Write 7 New Done Done true 7882 Acked 8881 Read 9 New Done Done true 7883 Acked 8882 Flush 0 New Done Done true 7884 Acked 8883 Read 4 New Done Done true 7885 Acked 8884 Write 7 New Done Done true 7886 Acked 8885 Write 2 New Done Done true 7887 Acked 8886 Read 9 New Done Done true 7888 Acked 8887 Write 1 New Done Done true 7889 Acked 8888 Flush 0 New Done Done true 7890 Acked 8889 Read 3 New Done Done true 7891 Acked 8890 Read 1 New Done Done true 7892 Acked 8891 Write 10 New Done Done true 7893 Acked 8892 Write 10 New Done Done true 7894 Acked 8893 Write 8 New Done Done true 7895 Acked 8894 Read 7 New Done Done true 7896 Acked 8895 Write 10 New Done Done true 7897 Acked 8896 Read 4 New Done Done true 7898 Acked 8897 Write 6 New Done Done true 7899 Acked 8898 Read 6 New Done Done true 7900 Acked 8899 Read 1 New Done Done true 7901 Acked 8900 Write 6 New Done Done true 7902 Acked 8901 Read 4 New Done Done true 7903 Acked 8902 Write 4 New Done Done true 7904 Acked 8903 Read 8 New Done Done true 7905 Acked 8904 Read 5 New Done Done true 7906 Acked 8905 Write 3 New Done Done true 7907 Acked 8906 Read 7 New Done Done true 7908 Acked 8907 Read 4 New Done Done true 7909 Acked 8908 Read 6 New Done Done true 7910 Acked 8909 Write 5 New Done Done true 7911 Acked 8910 Read 8 New Done Done true 7912 Acked 8911 Read 3 New Done Done true 7913 Acked 8912 Flush 0 New Done Done true 7914 Acked 8913 Write 3 New Done Done true 7915 Acked 8914 Write 7 New Done Done true 7916 Acked 8915 Read 1 New Done Done true 7917 Acked 8916 Read 5 New Done Done true 7918 Acked 8917 Read 8 New Done Done true 7919 Acked 8918 Read 7 New Done Done true 7920 Acked 8919 Read 6 New Done Done true 7921 Acked 8920 Read 2 New Done Done true 7922 Acked 8921 Write 6 New Done Done true 7923 Acked 8922 Write 9 New Done Done true 7924 Acked 8923 Read 6 New Done Done true 7925 Acked 8924 Read 2 New Done Done true 7926 Acked 8925 Read 10 New Done Done true 7927 Acked 8926 Write 6 New Done Done true 7928 Acked 8927 Write 3 New Done Done true 7929 Acked 8928 Read 5 New Done Done true 7930 Acked 8929 Read 5 New Done Done true 7931 Acked 8930 Read 1 New Done Done true 7932 Acked 8931 Read 10 New Done Done true 7933 Acked 8932 Read 3 New Done Done true 7934 Acked 8933 Read 1 New Done Done true 7935 Acked 8934 Read 7 New Done Done true 7936 Acked 8935 Read 10 New Done Done true 7937 Acked 8936 Read 4 New Done Done true 7938 Acked 8937 Write 9 New Done Done true 7939 Acked 8938 Write 8 New Done Done true 7940 Acked 8939 Read 2 New Done Done true 7941 Acked 8940 Read 6 New Done Done true 7942 Acked 8941 Write 7 New Done Done true 7943 Acked 8942 Write 7 New Done Done true 7944 Acked 8943 Read 2 New Done Done true 7945 Acked 8944 Read 4 New Done Done true 7946 Acked 8945 Write 1 New Done Done true 7947 Acked 8946 Read 6 New Done Done true 7948 Acked 8947 Write 3 New Done Done true 7949 Acked 8948 Read 2 New Done Done true 7950 Acked 8949 Read 9 New Done Done true 7951 Acked 8950 Write 8 New Done Done true 7952 Acked 8951 Read 10 New Done Done true 7953 Acked 8952 Write 7 New Done Done true 7954 Acked 8953 Flush 0 New Done Done true 7955 Acked 8954 Write 5 New Done Done true 7956 Acked 8955 Read 7 New Done Done true 7957 Acked 8956 Write 1 New Done Done true 7958 Acked 8957 Write 2 New Done Done true 7959 Acked 8958 Read 10 New Done Done true 7960 Acked 8959 Read 4 New Done Done true 7961 Acked 8960 Read 7 New Done Done true 7962 Acked 8961 Flush 0 New Done Done true 7963 Acked 8962 Flush 0 New Done Done true 7964 Acked 8963 Read 9 New Done Done true 7965 Acked 8964 Read 5 New Done Done true 7966 Acked 8965 Read 7 New Done Done true 7967 Acked 8966 Write 4 New Done Done true 7968 Acked 8967 Write 2 New Done Done true 7969 Acked 8968 Flush 0 New Done Done true 7970 Acked 8969 Read 2 New Done Done true 7971 Acked 8970 Read 9 New Done Done true 7972 Acked 8971 Write 2 New Done Done true 7973 Acked 8972 Read 10 New Done Done true 7974 Acked 8973 Read 6 New Done Done true 7975 Acked 8974 Read 5 New Done Done true 7976 Acked 8975 Write 1 New Done Done true 7977 Acked 8976 Write 10 New Done Done true 7978 Acked 8977 Read 8 New Done Done true 7979 Acked 8978 Write 9 New Done Done true 7980 Acked 8979 Read 2 New Done Done true 7981 Acked 8980 Flush 0 New Done Done true 7982 Acked 8981 Flush 0 New Done Done true 7983 Acked 8982 Read 1 New Done Done true 7984 Acked 8983 Read 8 New Done Done true 7985 Acked 8984 Read 4 New Done Done true 7986 Acked 8985 Write 4 New Done Done true 7987 Acked 8986 Read 8 New Done Done true 7988 Acked 8987 Write 2 New Done Done true 7989 Acked 8988 Write 7 New Done Done true 7990 Acked 8989 Read 4 New Done Done true 7991 Acked 8990 Read 6 New Done Done true 7992 Acked 8991 Write 2 New Done Done true 7993 Acked 8992 Write 2 New Done Done true 7994 Acked 8993 Flush 0 New Done Done true 7995 Acked 8994 Read 7 New Done Done true 7996 Acked 8995 Flush 0 New Done Done true 7997 Acked 8996 Write 9 New Done Done true 7998 Acked 8997 Read 6 New Done Done true 7999 Acked 8998 Flush 0 New Done Done true 8000 Acked 8999 Read 6 New Done Done true 8001 Acked 9000 Read 1 New Done Done true 8002 Acked 9001 Read 7 New Done Done true 8003 Acked 9002 Read 7 New Done Done true 8004 Acked 9003 Read 1 New Done Done true 8005 Acked 9004 Read 1 New Done Done true 8006 Acked 9005 Write 1 New Done Done true 8007 Acked 9006 Read 5 New Done Done true 8008 Acked 9007 Write 3 New Done Done true 8009 Acked 9008 Read 6 New Done Done true 8010 Acked 9009 Write 10 New Done Done true 8011 Acked 9010 Read 5 New Done Done true 8012 Acked 9011 Read 3 New Done Done true 8013 Acked 9012 Read 3 New Done Done true 8014 Acked 9013 Read 8 New Done Done true 8015 Acked 9014 Read 9 New Done Done true 8016 Acked 9015 Read 4 New Done Done true 8017 Acked 9016 Write 7 New Done Done true 8018 Acked 9017 Read 9 New Done Done true 8019 Acked 9018 Read 3 New Done Done true 8020 Acked 9019 Read 3 New Done Done true 8021 Acked 9020 Write 1 New Done Done true 8022 Acked 9021 Write 4 New Done Done true 8023 Acked 9022 Write 2 New Done Done true 8024 Acked 9023 Flush 0 New Done Done true 8025 Acked 9024 Write 9 New Done Done true 8026 Acked 9025 Read 10 New Done Done true 8027 Acked 9026 Write 6 New Done Done true 8028 Acked 9027 Read 8 New Done Done true 8029 Acked 9028 Flush 0 New Done Done true 8030 Acked 9029 Read 5 New Done Done true 8031 Acked 9030 Flush 0 New Done Done true 8032 Acked 9031 Read 4 New Done Done true 8033 Acked 9032 Read 10 New Done Done true 8034 Acked 9033 Read 6 New Done Done true 8035 Acked 9034 Flush 0 New Done Done true 8036 Acked 9035 Read 10 New Done Done true 8037 Acked 9036 Read 10 New Done Done true 8038 Acked 9037 Read 5 New Done Done true 8039 Acked 9038 Read 7 New Done Done true 8040 Acked 9039 Flush 0 New Done Done true 8041 Acked 9040 Read 2 New Done Done true 8042 Acked 9041 Write 5 New Done Done true 8043 Acked 9042 Write 3 New Done Done true 8044 Acked 9043 Read 2 New Done Done true 8045 Acked 9044 Write 4 New Done Done true 8046 Acked 9045 Read 6 New Done Done true 8047 Acked 9046 Write 6 New Done Done true 8048 Acked 9047 Read 3 New Done Done true 8049 Acked 9048 Write 10 New Done Done true 8050 Acked 9049 Write 7 New Done Done true 8051 Acked 9050 Flush 0 New Done Done true 8052 Acked 9051 Read 3 New Done Done true 8053 Acked 9052 Read 7 New Done Done true 8054 Acked 9053 Read 7 New Done Done true 8055 Acked 9054 Write 1 New Done Done true 8056 Acked 9055 Write 8 New Done Done true 8057 Acked 9056 Read 8 New Done Done true 8058 Acked 9057 Write 10 New Done Done true 8059 Acked 9058 Write 9 New Done Done true 8060 Acked 9059 Read 6 New Done Done true 8061 Acked 9060 Write 2 New Done Done true 8062 Acked 9061 Write 7 New Done Done true 8063 Acked 9062 Write 9 New Done Done true 8064 Acked 9063 Flush 0 New Done Done true 8065 Acked 9064 Read 4 New Done Done true 8066 Acked 9065 Read 6 New Done Done true 8067 Acked 9066 Read 8 New Done Done true 8068 Acked 9067 Read 5 New Done Done true 8069 Acked 9068 Read 7 New Done Done true 8070 Acked 9069 Read 3 New Done Done true 8071 Acked 9070 Read 2 New Done Done true 8072 Acked 9071 Write 4 New Done Done true 8073 Acked 9072 Write 6 New Done Done true 8074 Acked 9073 Read 4 New Done Done true 8075 Acked 9074 Write 3 New Done Done true 8076 Acked 9075 Read 4 New Done Done true 8077 Acked 9076 Read 1 New Done Done true 8078 Acked 9077 Read 3 New Done Done true 8079 Acked 9078 Read 8 New Done Done true 8080 Acked 9079 Read 10 New Done Done true 8081 Acked 9080 Write 5 New Done Done true 8082 Acked 9081 Read 5 New Done Done true 8083 Acked 9082 Read 10 New Done Done true 8084 Acked 9083 Read 7 New Done Done true 8085 Acked 9084 Write 6 New Done Done true 8086 Acked 9085 Read 9 New Done Done true 8087 Acked 9086 Read 9 New Done Done true 8088 Acked 9087 Flush 0 New Done Done true 8089 Acked 9088 Read 2 New Done Done true 8090 Acked 9089 Flush 0 New Done Done true 8091 Acked 9090 Read 4 New Done Done true 8092 Acked 9091 Write 7 New Done Done true 8093 Acked 9092 Read 8 New Done Done true 8094 Acked 9093 Write 8 New Done Done true 8095 Acked 9094 Write 3 New Done Done true 8096 Acked 9095 Read 7 New Done Done true 8097 Acked 9096 Write 4 New Done Done true 8098 Acked 9097 Write 10 New Done Done true 8099 Acked 9098 Read 6 New Done Done true 8100 Acked 9099 Write 9 New Done Done true 8101 Acked 9100 Write 3 New Done Done true 8102 Acked 9101 Read 4 New Done Done true 8103 Acked 9102 Flush 0 New Done Done true 8104 Acked 9103 Read 9 New Done Done true 8105 Acked 9104 Flush 0 New Done Done true 8106 Acked 9105 Read 7 New Done Done true 8107 Acked 9106 Write 3 New Done Done true 8108 Acked 9107 Read 2 New Done Done true 8109 Acked 9108 Write 8 New Done Done true 8110 Acked 9109 Read 5 New Done Done true 8111 Acked 9110 Read 8 New Done Done true 8112 Acked 9111 Read 6 New Done Done true 8113 Acked 9112 Read 5 New Done Done true 8114 Acked 9113 Write 1 New Done Done true 8115 Acked 9114 Write 3 New Done Done true 8116 Acked 9115 Flush 0 New Done Done true 8117 Acked 9116 Read 10 New Done Done true 8118 Acked 9117 Read 9 New Done Done true 8119 Acked 9118 Read 6 New Done Done true 8120 Acked 9119 Read 1 New Done Done true 8121 Acked 9120 Flush 0 New Done Done true 8122 Acked 9121 Read 4 New Done Done true 8123 Acked 9122 Read 10 New Done Done true 8124 Acked 9123 Read 8 New Done Done true 8125 Acked 9124 Read 3 New Done Done true 8126 Acked 9125 Write 7 New Done Done true 8127 Acked 9126 Write 4 New Done Done true 8128 Acked 9127 Write 3 New Done Done true 8129 Acked 9128 Flush 0 New Done Done true 8130 Acked 9129 Read 2 New Done Done true 8131 Acked 9130 Read 7 New Done Done true 8132 Acked 9131 Write 10 New Done Done true 8133 Acked 9132 Write 9 New Done Done true 8134 Acked 9133 Write 4 New Done Done true 8135 Acked 9134 Read 3 New Done Done true 8136 Acked 9135 Write 3 New Done Done true 8137 Acked 9136 Read 9 New Done Done true 8138 Acked 9137 Write 1 New Done Done true 8139 Acked 9138 Write 4 New Done Done true 8140 Acked 9139 Read 3 New Done Done true 8141 Acked 9140 Read 5 New Done Done true 8142 Acked 9141 Write 3 New Done Done true 8143 Acked 9142 Read 5 New Done Done true 8144 Acked 9143 Write 2 New Done Done true 8145 Acked 9144 Write 3 New Done Done true 8146 Acked 9145 Read 10 New Done Done true 8147 Acked 9146 Flush 0 New Done Done true 8148 Acked 9147 Write 2 New Done Done true 8149 Acked 9148 Write 10 New Done Done true 8150 Acked 9149 Read 5 New Done Done true 8151 Acked 9150 Write 4 New Done Done true 8152 Acked 9151 Write 2 New Done Done true 8153 Acked 9152 Flush 0 New Done Done true STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 8851 9152 9152 Downstairs last five completed: 8851 8850 8849 8848 8847 Upstairs last five completed: 8153 8152 8151 8150 8149 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.776952296Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.777101717Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779178944Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779209854Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779233503Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 8851","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779563654Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779598881Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779622839Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779645217Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:54.779667114Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 9152 9151 9150 9149 9148 Upstairs last five completed: 8153 8152 8151 8150 8149 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(2) stop 2 Jan 02 22:35:58.394 INFO request completed, latency_us: 468, response_code: 204, uri: /stop/cid/2, method: POST, req_id: bbc11552-cbe0-44de-8e81-e13048dd100b, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 [8830] Got stop action so:false kr:false Jan 02 22:35:58.395 INFO request completed, latency_us: 311, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9cb665d2-f18e-40d6-80a2-0ed20a742029, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:58.39561262Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:58.395713974Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:58.395819971Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:58.395874044Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:58.395917847Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:58.395959954Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:59.397758345Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:59.397828208Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:59.397866928Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:35:59.397894085Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:59.397914481Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:59.398011264Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:35:59.398054187Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:00.400008904Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:00.400081278Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:00.400110264Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:00.400136665Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:00.400156997Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:00.40025203Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:00.400374129Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:01.401912262Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:01.401992962Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:01.402022344Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:01.40204633Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:01.402072716Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:01.402092974Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:01.402115251Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:01.402137476Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:36:02.397 INFO request completed, latency_us: 252, response_code: 200, uri: /state/cid/2, method: GET, req_id: af28fc65-fdc0-420f-9e79-a379191a687b, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 389 len 20480 data: 6 5 7 8 9 002/300 Write block 4978 len 16384 data: 4 7 6 6 003/300 Flush 004/300 Read block 1463 len 24576 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:02.403289124Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:02.403357229Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:02.403385867Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:02.403492026Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:02.40351891Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 4 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:02.403619212Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:02.403737333Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 005/300 Read block 4076 len 24576 006/300 Read block 4866 len 40960 007/300 Write block 4690 len 36864 data: 3 2 3 4 5 8 8 8 6 008/300 Write block 1323 len 16384 data: 8 7 7 7 009/300 Read block 3350 len 12288 010/300 Read block 1659 len 28672 011/300 Read block 2597 len 16384 012/300 Flush 013/300 Write block 4065 len 40960 data: 3 3 3 2 2 2 3 3 4 5 014/300 Write block 3490 len 28672 data: 3 3 3 3 4 3 4 015/300 Flush 016/300 Write block 1212 len 40960 data: 6 6 6 5 4 6 6 5 7 7 017/300 Write block 2082 len 4096 data: 4 018/300 Read block 403 len 36864 019/300 Write block 1036 len 24576 data: 6 5 5 7 8 6 020/300 Read block 3706 len 12288 021/300 Write block 1951 len 12288 data: 3 2 3 022/300 Write block 4093 len 4096 data: 8 023/300 Flush 024/300 Read block 3678 len 28672 025/300 Write block 4157 len 16384 data: 5 3 3 3 026/300 Write block 4126 len 4096 data: 8 027/300 Read block 1154 len 16384 028/300 Write block 2870 len 4096 data: 6 029/300 Read block 1250 len 12288 030/300 Read block 4025 len 16384 031/300 Write block 4683 len 16384 data: 4 4 4 5 032/300 Read block 4921 len 4096 033/300 Write block 3349 len 32768 data: 7 6 5 4 4 5 5 4 034/300 Write block 4711 len 12288 data: 4 3 3 035/300 Write block 3596 len 40960 data: 4 4 5 5 6 6 8 7 6 7 036/300 Read block 4219 len 8192 037/300 Read block 516 len 8192 038/300 Flush 039/300 Read block 3884 len 32768 040/300 Write block 858 len 32768 data: 5 5 6 5 5 4 6 5 041/300 Write block 1688 len 12288 data: 11 8 8 042/300 Read block 1848 len 40960 043/300 Write block 285 len 12288 data: 7 9 9 044/300 Read block 1722 len 16384 045/300 Read block 2977 len 8192 046/300 Read block 2111 len 8192 047/300 Read block 2164 len 40960 048/300 Read block 4362 len 20480 049/300 Write block 2484 len 12288 data: 6 5 6 050/300 Read block 3221 len 8192 051/300 Read block 1362 len 32768 052/300 Write block 3214 len 28672 data: 8 10 6 6 6 5 6 053/300 Write block 4231 len 4096 data: 7 054/300 Read block 44 len 8192 055/300 Read block 1667 len 24576 056/300 Read block 3447 len 28672 057/300 Write block 2975 len 28672 data: 5 4 4 4 4 3 3 058/300 Write block 2587 len 40960 data: 4 4 4 4 4 3 3 2 3 5 059/300 Read block 1011 len 16384 060/300 Write block 1448 len 8192 data: 5 4 061/300 Read block 4844 len 36864 062/300 Read block 2264 len 28672 063/300 Write block 1578 len 40960 data: 2 3 3 6 6 6 6 9 6 5 064/300 Write block 2053 len 4096 data: 7 065/300 Flush 066/300 Read block 542 len 8192 067/300 Write block 3620 len 20480 data: 6 6 6 7 6 068/300 Read block 4462 len 32768 069/300 Write block 4726 len 16384 data: 6 5 5 5 070/300 Read block 1924 len 28672 071/300 Write block 4009 len 4096 data: 12 072/300 Write block 1694 len 28672 data: 8 7 6 6 6 4 4 073/300 Read block 2933 len 12288 074/300 Flush 075/300 Write block 2869 len 24576 data: 7 7 7 5 7 6 076/300 Read block 3182 len 28672 077/300 Write block 4727 len 20480 data: 6 6 6 8 8 078/300 Write block 265 len 40960 data: 6 6 9 10 9 9 10 9 9 8 079/300 Write block 4137 len 20480 data: 4 4 4 4 5 080/300 Read block 2349 len 28672 081/300 Write block 2653 len 32768 data: 9 7 7 6 5 4 4 4 082/300 Read block 1543 len 24576 083/300 Write block 803 len 40960 data: 6 4 5 5 5 6 5 4 5 6 084/300 Read block 374 len 16384 085/300 Read block 187 len 20480 086/300 Flush 087/300 Write block 1365 len 16384 data: 4 4 3 3 088/300 Read block 3276 len 32768 089/300 Flush 090/300 Read block 2299 len 40960 091/300 Read block 2675 len 20480 092/300 Flush 093/300 Read block 337 len 20480 094/300 Write block 4631 len 36864 data: 6 6 8 6 6 6 8 6 7 095/300 Write block 4088 len 20480 data: 7 7 9 10 10 096/300 Write block 1604 len 40960 data: 4 5 4 3 4 4 5 5 5 4 097/300 Read block 3784 len 20480 098/300 Write block 1556 len 12288 data: 8 7 9 099/300 Write block 1500 len 8192 data: 8 6 100/300 Flush 101/300 Read block 1414 len 32768 102/300 Read block 1587 len 8192 103/300 Read block 4021 len 8192 104/300 Write block 2905 len 20480 data: 5 7 5 5 4 105/300 Write block 4197 len 8192 data: 3 3 106/300 Read block 2548 len 20480 107/300 Flush 108/300 Read block 406 len 28672 109/300 Write block 1622 len 28672 data: 4 4 6 8 8 9 6 110/300 Flush 111/300 Read block 2354 len 4096 112/300 Write block 1648 len 40960 data: 3 3 4 5 5 7 7 9 9 8 113/300 Write block 1260 len 8192 data: 9 11 114/300 Write block 3081 len 12288 data: 6 7 8 115/300 Read block 2100 len 28672 116/300 Read block 4157 len 32768 117/300 Read block 2373 len 40960 118/300 Read block 23 len 24576 119/300 Read block 1481 len 12288 120/300 Read block 147 len 28672 121/300 Read block 3686 len 8192 122/300 Flush 123/300 Flush 124/300 Read block 4961 len 4096 125/300 Flush 126/300 Read block 564 len 4096 127/300 Write block 3483 len 32768 data: 7 6 5 4 4 3 4 4 128/300 Read block 3152 len 12288 129/300 Read block 1371 len 4096 130/300 Read block 2397 len 12288 131/300 Read block 2641 len 16384 132/300 Write block 3819 len 28672 data: 6 6 6 4 6 4 4 133/300 Read block 3617 len 28672 134/300 Read block 42 len 32768 135/300 Read block 4447 len 36864 136/300 Write block 2223 len 36864 data: 5 5 6 8 7 6 6 5 5 137/300 Write block 3060 len 20480 data: 11 10 9 9 9 138/300 Write block 4849 len 20480 data: 3 2 2 3 3 139/300 Read block 632 len 32768 140/300 Write block 4059 len 8192 data: 3 3 141/300 Write block 361 len 40960 data: 4 6 8 10 7 10 11 9 7 5 142/300 Read block 309 len 12288 143/300 Flush 144/300 Read block 3881 len 24576 145/300 Write block 1971 len 24576 data: 8 8 6 6 5 3 146/300 Read block 4134 len 32768 147/300 Write block 4188 len 20480 data: 4 4 3 3 4 148/300 Read block 4749 len 12288 149/300 Write block 970 len 36864 data: 6 6 6 6 5 6 7 6 4 150/300 Write block 3789 len 28672 data: 9 8 10 11 8 8 6 151/300 Write block 4205 len 40960 data: 6 5 6 5 6 4 4 4 6 7 152/300 Write block 3743 len 36864 data: 4 4 5 5 5 6 4 4 3 153/300 Write block 4588 len 8192 data: 3 5 154/300 Write block 451 len 8192 data: 6 6 155/300 Read block 4283 len 12288 156/300 Write block 2382 len 12288 data: 3 3 4 157/300 Write block 70 len 8192 data: 2 2 158/300 Read block 2146 len 36864 159/300 Flush 160/300 Read block 4025 len 16384 161/300 Read block 4572 len 8192 162/300 Read block 4457 len 8192 163/300 Write block 2131 len 16384 data: 5 5 5 5 164/300 Read block 1152 len 36864 165/300 Read block 1961 len 40960 166/300 Read block 110 len 8192 167/300 Read block 4522 len 32768 168/300 Flush 169/300 Write block 44 len 24576 data: 4 5 5 6 7 7 170/300 Write block 4379 len 24576 data: 6 6 5 5 7 7 171/300 Read block 991 len 36864 172/300 Read block 2085 len 12288 173/300 Flush 174/300 Write block 639 len 32768 data: 4 4 4 4 5 5 5 5 175/300 Write block 2817 len 28672 data: 5 7 7 8 11 12 13 176/300 Flush 177/300 Read block 883 len 12288 178/300 Write block 416 len 28672 data: 7 7 9 9 8 9 10 179/300 Read block 660 len 20480 180/300 Write block 1510 len 40960 data: 9 7 5 5 4 4 4 4 5 6 181/300 Read block 984 len 36864 182/300 Write block 3378 len 40960 data: 6 6 6 5 4 3 5 4 4 4 183/300 Read block 1128 len 4096 184/300 Read block 880 len 4096 185/300 Read block 42 len 12288 186/300 Read block 866 len 16384 187/300 Read block 2393 len 24576 188/300 Read block 3595 len 16384 189/300 Read block 996 len 8192 190/300 Write block 1461 len 8192 data: 8 6 191/300 Write block 185 len 8192 data: 3 3 192/300 Write block 2796 len 24576 data: 2 5 6 9 9 8 193/300 Write block 4493 len 20480 data: 6 6 5 5 5 194/300 Read block 2501 len 36864 195/300 Write block 3043 len 40960 data: 4 6 6 7 7 9 6 6 5 5 196/300 Write block 3613 len 28672 data: 7 7 6 6 5 5 5 197/300 Write block 2121 len 36864 data: 9 9 8 6 5 6 5 4 5 198/300 Write block 3872 len 4096 data: 5 199/300 Write block 3599 len 24576 data: 6 7 7 9 8 7 200/300 Write block 959 len 12288 data: 4 4 4 201/300 Read block 4358 len 28672 202/300 Read block 63 len 8192 203/300 Write block 3298 len 16384 data: 5 6 7 7 204/300 Read block 2446 len 32768 205/300 Read block 762 len 16384 206/300 Read block 2614 len 32768 207/300 Read block 926 len 20480 208/300 Read block 4848 len 40960 209/300 Read block 2934 len 4096 210/300 Flush 211/300 Flush 212/300 Read block 1687 len 4096 213/300 Read block 275 len 16384 214/300 Read block 3369 len 36864 215/300 Read block 1542 len 40960 216/300 Read block 1432 len 40960 217/300 Read block 4096 len 28672 218/300 Read block 2932 len 20480 219/300 Read block 3429 len 16384 220/300 Write block 1190 len 12288 data: 7 7 7 221/300 Read block 4487 len 28672 222/300 Write block 2100 len 12288 data: 4 3 3 223/300 Read block 3178 len 28672 224/300 Read block 2512 len 12288 225/300 Read block 763 len 4096 226/300 Write block 4165 len 12288 data: 5 5 4 227/300 Read block 2836 len 40960 228/300 Read block 499 len 4096 229/300 Write block 4713 len 28672 data: 4 2 3 4 4 4 4 230/300 Write block 4239 len 20480 data: 5 6 7 5 5 231/300 Write block 3959 len 28672 data: 8 9 7 8 7 7 5 232/300 Flush 233/300 Write block 4319 len 32768 data: 5 7 8 8 8 8 8 9 234/300 Write block 4387 len 24576 data: 5 5 5 4 4 3 235/300 Write block 1959 len 28672 data: 4 4 4 3 5 6 4 236/300 Read block 1250 len 4096 237/300 Flush 238/300 Read block 3393 len 20480 239/300 Flush 240/300 Read block 287 len 32768 241/300 Write block 2207 len 36864 data: 4 5 6 6 6 5 5 4 4 242/300 Flush 243/300 Read block 957 len 12288 244/300 Read block 3464 len 32768 245/300 Write block 484 len 20480 data: 6 6 5 5 5 246/300 Write block 4294 len 28672 data: 6 7 5 4 3 3 2 247/300 Write block 1097 len 12288 data: 5 6 5 248/300 Write block 1281 len 4096 data: 7 249/300 Write block 4973 len 16384 data: 3 5 5 6 250/300 Write block 2295 len 24576 data: 4 5 5 6 6 7 251/300 Read block 706 len 20480 252/300 Read block 3264 len 16384 253/300 Write block 952 len 36864 data: 5 6 5 5 6 5 4 5 5 254/300 Write block 842 len 8192 data: 4 5 255/300 Read block 3196 len 24576 256/300 Write block 2013 len 32768 data: 5 4 5 5 7 6 5 6 257/300 Write block 4608 len 32768 data: 9 10 9 9 9 7 8 7 258/300 Write block 2383 len 12288 data: 4 5 7 259/300 Read block 3050 len 40960 260/300 Write block 2570 len 4096 data: 4 261/300 Read block 2222 len 20480 262/300 Read block 4627 len 24576 263/300 Read block 1688 len 32768 264/300 Write block 4565 len 32768 data: 6 7 7 7 7 8 5 6 265/300 Read block 2394 len 12288 266/300 Read block 676 len 4096 267/300 Read block 4196 len 40960 268/300 Read block 924 len 36864 269/300 Read block 2163 len 24576 270/300 Write block 4322 len 32768 data: 9 9 9 9 10 9 8 7 271/300 Read block 4816 len 28672 272/300 Read block 1840 len 32768 273/300 Read block 1198 len 24576 274/300 Read block 2781 len 16384 275/300 Read block 2856 len 36864 276/300 Write block 2584 len 32768 data: 6 5 4 5 5 5 5 5 277/300 Write block 1611 len 16384 data: 6 6 5 3 278/300 Read block 133 len 8192 279/300 Write block 3514 len 32768 data: 5 6 5 5 5 4 3 2 280/300 Flush 281/300 Read block 243 len 4096 282/300 Read block 1124 len 8192 283/300 Read block 4372 len 36864 284/300 Flush 285/300 Flush 286/300 Write block 1616 len 24576 data: 4 5 4 3 3 4 287/300 Write block 1011 len 40960 data: 6 6 6 7 5 5 5 5 5 4 288/300 Write block 3666 len 40960 data: 7 7 7 5 5 4 4 5 7 6 289/300 Write block 4063 len 32768 data: 3 3 4 4 4 3 3 3 290/300 Read block 3519 len 4096 291/300 Write block 2744 len 8192 data: 4 4 292/300 Write block 4414 len 36864 data: 4 4 4 5 7 8 7 6 7 293/300 Write block 1427 len 28672 data: 5 6 5 4 4 4 3 294/300 Write block 4386 len 8192 data: 8 6 295/300 Flush 296/300 Write block 967 len 28672 data: 6 6 6 7 7 7 7 297/300 Read block 51 len 40960 298/300 Read block 434 len 40960 299/300 Write block 2784 len 16384 data: 6 6 6 7 300/300 Read block 1966 len 4096 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:36:03.022 INFO request completed, latency_us: 295, response_code: 204, uri: /start/cid/2, method: POST, req_id: ca572e85-0d90-49c7-8e9c-8a9dee99ff2e, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 8154 Acked 9153 Write 5 Done Done New true 8155 Acked 9154 Write 4 Done Done New true 8156 Acked 9155 Flush 0 Done Done New true 8157 Acked 9156 Read 6 Done Done New true 8158 Acked 9157 Read 6 Done Done New false 8159 Acked 9158 Read 10 Done Done New false 8160 Acked 9159 Write 9 Done Done New false 8161 Acked 9160 Write 4 Done Done New false 8162 Acked 9161 Read 3 Done Done New false 8163 Acked 9162 Read 7 Done Done New false 8164 Acked 9163 Read 4 Done Done New false 8165 Acked 9164 Flush 0 Done Done New false 8166 Acked 9165 Write 10 Done Done New false 8167 Acked 9166 Write 7 Done Done New false 8168 Acked 9167 Flush 0 Done Done New false 8169 Acked 9168 Write 10 Done Done New false 8170 Acked 9169 Write 1 Done Done New false 8171 Acked 9170 Read 9 Done Done New false 8172 Acked 9171 Write 6 Done Done New false 8173 Acked 9172 Read 3 Done Done New false 8174 Acked 9173 Write 3 Done Done New false 8175 Acked 9174 Write 1 Done Done New false 8176 Acked 9175 Flush 0 Done Done New false 8177 Acked 9176 Read 7 Done Done New false 8178 Acked 9177 Write 4 Done Done New false 8179 Acked 9178 Write 1 Done Done New false 8180 Acked 9179 Read 4 Done Done New false 8181 Acked 9180 Write 1 Done Done New false 8182 Acked 9181 Read 3 Done Done New false 8183 Acked 9182 Read 4 Done Done New false 8184 Acked 9183 Write 4 Done Done New false 8185 Acked 9184 Read 1 Done Done New false 8186 Acked 9185 Write 8 Done Done New false 8187 Acked 9186 Write 3 Done Done New false 8188 Acked 9187 Write 10 Done Done New false 8189 Acked 9188 Read 2 Done Done New false 8190 Acked 9189 Read 2 Done Done New false 8191 Acked 9190 Flush 0 Done Done New false 8192 Acked 9191 Read 8 Done Done New false 8193 Acked 9192 Write 8 Done Done New false 8194 Acked 9193 Write 3 Done Done New false 8195 Acked 9194 Read 10 Done Done New false 8196 Acked 9195 Write 3 Done Done New false 8197 Acked 9196 Read 4 Done Done New false 8198 Acked 9197 Read 2 Done Done New false 8199 Acked 9198 Read 2 Done Done New false 8200 Acked 9199 Read 10 Done Done New false 8201 Acked 9200 Read 5 Done Done New false 8202 Acked 9201 Write 3 Done Done New false 8203 Acked 9202 Read 2 Done Done New false 8204 Acked 9203 Read 8 Done Done New false 8205 Acked 9204 Write 7 Done Done New false 8206 Acked 9205 Write 1 Done Done New false 8207 Acked 9206 Read 2 Done Done New false 8208 Acked 9207 Read 6 Done Done New false 8209 Acked 9208 Flush 0 Done Done New false 8210 Acked 9209 Read 7 Done Done New false 8211 Acked 9210 Write 7 Done Done New false 8212 Acked 9211 Write 10 Done Done New false 8213 Acked 9212 Read 4 Done Done New false 8214 Acked 9213 Write 2 Done Done New false 8215 Acked 9214 Read 9 Done Done New false 8216 Acked 9215 Read 7 Done Done New false 8217 Acked 9216 Write 10 Done Done New false 8218 Acked 9217 Write 1 Done Done New false 8219 Acked 9218 Flush 0 Done Done New false 8220 Acked 9219 Read 2 Done Done New false 8221 Acked 9220 Write 5 Done Done New false 8222 Acked 9221 Read 8 Done Done New false 8223 Acked 9222 Write 4 Done Done New false 8224 Acked 9223 Read 7 Done Done New false 8225 Acked 9224 Write 1 Done Done New false 8226 Acked 9225 Write 7 Done Done New false 8227 Acked 9226 Read 3 Done Done New false 8228 Acked 9227 Flush 0 Done Done New false 8229 Acked 9228 Write 6 Done Done New false 8230 Acked 9229 Read 7 Done Done New false 8231 Acked 9230 Write 5 Done Done New false 8232 Acked 9231 Write 10 Done Done New false 8233 Acked 9232 Write 5 Done Done New false 8234 Acked 9233 Read 7 Done Done New false 8235 Acked 9234 Write 8 Done Done New false 8236 Acked 9235 Read 6 Done Done New false 8237 Acked 9236 Write 10 Done Done New false 8238 Acked 9237 Read 4 Done Done New false 8239 Acked 9238 Read 5 Done Done New false 8240 Acked 9239 Flush 0 Done Done New false 8241 Acked 9240 Write 4 Done Done New false 8242 Acked 9241 Read 8 Done Done New false 8243 Acked 9242 Flush 0 Done Done New false 8244 Acked 9243 Read 10 Done Done New false 8245 Acked 9244 Read 5 Done Done New false 8246 Acked 9245 Flush 0 Done Done New false 8247 Acked 9246 Read 5 Done Done New false 8248 Acked 9247 Write 9 Done Done New false 8249 Acked 9248 Write 5 Done Done New false 8250 Acked 9249 Write 10 Done Done New false 8251 Acked 9250 Read 5 Done Done New false 8252 Acked 9251 Write 3 Done Done New false 8253 Acked 9252 Write 2 Done Done New false 8254 Acked 9253 Flush 0 Done Done New false 8255 Acked 9254 Read 8 Done Done New false 8256 Acked 9255 Read 2 Done Done New false 8257 Acked 9256 Read 2 Done Done New false 8258 Acked 9257 Write 5 Done Done New false 8259 Acked 9258 Write 2 Done Done New false 8260 Acked 9259 Read 5 Done Done New false 8261 Acked 9260 Flush 0 Done Done New false 8262 Acked 9261 Read 7 Done Done New false 8263 Acked 9262 Write 7 Done Done New false 8264 Acked 9263 Flush 0 Done Done New false 8265 Acked 9264 Read 1 Done Done New false 8266 Acked 9265 Write 10 Done Done New false 8267 Acked 9266 Write 2 Done Done New false 8268 Acked 9267 Write 3 Done Done New false 8269 Acked 9268 Read 7 Done Done New false 8270 Acked 9269 Read 8 Done Done New false 8271 Acked 9270 Read 10 Done Done New false 8272 Acked 9271 Read 6 Done Done New false 8273 Acked 9272 Read 3 Done Done New false 8274 Acked 9273 Read 7 Done Done New false 8275 Acked 9274 Read 2 Done Done New false 8276 Acked 9275 Flush 0 Done Done New false 8277 Acked 9276 Flush 0 Done Done New false 8278 Acked 9277 Read 1 Done Done New false 8279 Acked 9278 Flush 0 Done Done New false 8280 Acked 9279 Read 1 Done Done New false 8281 Acked 9280 Write 8 Done Done New false 8282 Acked 9281 Read 3 Done Done New false 8283 Acked 9282 Read 1 Done Done New false 8284 Acked 9283 Read 3 Done Done New false 8285 Acked 9284 Read 4 Done Done New false 8286 Acked 9285 Write 7 Done Done New false 8287 Acked 9286 Read 7 Done Done New false 8288 Acked 9287 Read 8 Done Done New false 8289 Acked 9288 Read 9 Done Done New false 8290 Acked 9289 Write 9 Done Done New false 8291 Acked 9290 Write 5 Done Done New false 8292 Acked 9291 Write 5 Done Done New false 8293 Acked 9292 Read 8 Done Done New false 8294 Acked 9293 Write 2 Done Done New false 8295 Acked 9294 Write 10 Done Done New false 8296 Acked 9295 Read 3 Done Done New false 8297 Acked 9296 Flush 0 Done Done New false 8298 Acked 9297 Read 6 Done Done New false 8299 Acked 9298 Write 6 Done Done New false 8300 Acked 9299 Read 8 Done Done New false 8301 Acked 9300 Write 5 Done Done New false 8302 Acked 9301 Read 3 Done Done New false 8303 Acked 9302 Write 9 Done Done New false 8304 Acked 9303 Write 7 Done Done New false 8305 Acked 9304 Write 10 Done Done New false 8306 Acked 9305 Write 9 Done Done New false 8307 Acked 9306 Write 2 Done Done New false 8308 Acked 9307 Write 2 Done Done New false 8309 Acked 9308 Read 3 Done Done New false 8310 Acked 9309 Write 3 Done Done New false 8311 Acked 9310 Write 2 Done Done New false 8312 Acked 9311 Read 9 Done Done New false 8313 Acked 9312 Flush 0 Done Done New false 8314 Acked 9313 Read 4 Done Done New false 8315 Acked 9314 Read 2 Done Done New false 8316 Acked 9315 Read 2 Done Done New false 8317 Acked 9316 Write 4 Done Done New false 8318 Acked 9317 Read 9 Done Done New false 8319 Acked 9318 Read 10 Done Done New false 8320 Acked 9319 Read 2 Done Done New false 8321 Acked 9320 Read 8 Done Done New false 8322 Acked 9321 Flush 0 Done Done New false 8323 Acked 9322 Write 6 Done Done New false 8324 Acked 9323 Write 6 Done Done New false 8325 Acked 9324 Read 9 Done Done New false 8326 Acked 9325 Read 3 Done Done New false 8327 Acked 9326 Flush 0 Done Done New false 8328 Acked 9327 Write 8 Done Done New false 8329 Acked 9328 Write 7 Done Done New false 8330 Acked 9329 Flush 0 Done Done New false 8331 Acked 9330 Read 3 Done Done New false 8332 Acked 9331 Write 7 Done Done New false 8333 Acked 9332 Read 5 Done Done New false 8334 Acked 9333 Write 10 Done Done New false 8335 Acked 9334 Read 9 Done Done New false 8336 Acked 9335 Write 10 Done Done New false 8337 Acked 9336 Read 1 Done Done New false 8338 Acked 9337 Read 1 Done Done New false 8339 Acked 9338 Read 3 Done Done New false 8340 Acked 9339 Read 4 Done Done New false 8341 Acked 9340 Read 6 Done Done New false 8342 Acked 9341 Read 4 Done Done New false 8343 Acked 9342 Read 2 Done Done New false 8344 Acked 9343 Write 2 Done Done New false 8345 Acked 9344 Write 2 Done Done New false 8346 Acked 9345 Write 6 Done Done New false 8347 Acked 9346 Write 5 Done Done New false 8348 Acked 9347 Read 9 Done Done New false 8349 Acked 9348 Write 10 Done Done New false 8350 Acked 9349 Write 7 Done Done New false 8351 Acked 9350 Write 9 Done Done New false 8352 Acked 9351 Write 1 Done Done New false 8353 Acked 9352 Write 6 Done Done New false 8354 Acked 9353 Write 3 Done Done New false 8355 Acked 9354 Read 7 Done Done New false 8356 Acked 9355 Read 2 Done Done New false 8357 Acked 9356 Write 4 Done Done New false 8358 Acked 9357 Read 8 Done Done New false 8359 Acked 9358 Read 4 Done Done New false 8360 Acked 9359 Read 8 Done Done New false 8361 Acked 9360 Read 5 Done Done New false 8362 Acked 9361 Read 10 Done Done New false 8363 Acked 9362 Read 1 Done Done New false 8364 Acked 9363 Flush 0 Done Done New false 8365 Acked 9364 Flush 0 Done Done New false 8366 Acked 9365 Read 1 Done Done New false 8367 Acked 9366 Read 4 Done Done New false 8368 Acked 9367 Read 9 Done Done New false 8369 Acked 9368 Read 10 Done Done New false 8370 Acked 9369 Read 10 Done Done New false 8371 Acked 9370 Read 7 Done Done New false 8372 Acked 9371 Read 5 Done Done New false 8373 Acked 9372 Read 4 Done Done New false 8374 Acked 9373 Write 3 Done Done New false 8375 Acked 9374 Read 7 Done Done New false 8376 Acked 9375 Write 3 Done Done New false 8377 Acked 9376 Read 7 Done Done New false 8378 Acked 9377 Read 3 Done Done New false 8379 Acked 9378 Read 1 Done Done New false 8380 Acked 9379 Write 3 Done Done New false 8381 Acked 9380 Read 10 Done Done New false 8382 Acked 9381 Read 1 Done Done New false 8383 Acked 9382 Write 7 Done Done New false 8384 Acked 9383 Write 5 Done Done New false 8385 Acked 9384 Write 7 Done Done New false 8386 Acked 9385 Flush 0 Done Done New false 8387 Acked 9386 Write 8 Done Done New false 8388 Acked 9387 Write 6 Done Done New false 8389 Acked 9388 Write 7 Done Done New false 8390 Acked 9389 Read 1 Done Done New false 8391 Acked 9390 Flush 0 Done Done New false 8392 Acked 9391 Read 5 Done Done New false 8393 Acked 9392 Flush 0 Done Done New false 8394 Acked 9393 Read 8 Done Done New false 8395 Acked 9394 Write 9 Done Done New false 8396 Acked 9395 Flush 0 Done Done New false 8397 Acked 9396 Read 3 Done Done New false 8398 Acked 9397 Read 8 Done Done New false 8399 Acked 9398 Write 5 Done Done New false 8400 Acked 9399 Write 7 Done Done New false 8401 Acked 9400 Write 3 Done Done New false 8402 Acked 9401 Write 1 Done Done New false 8403 Acked 9402 Write 4 Done Done New false 8404 Acked 9403 Write 6 Done Done New false 8405 Acked 9404 Read 5 Done Done New false 8406 Acked 9405 Read 4 Done Done New false 8407 Acked 9406 Write 9 Done Done New false 8408 Acked 9407 Write 2 Done Done New false 8409 Acked 9408 Read 6 Done Done New false 8410 Acked 9409 Write 8 Done Done New false 8411 Acked 9410 Write 8 Done Done New false 8412 Acked 9411 Write 3 Done Done New false 8413 Acked 9412 Read 10 Done Done New false 8414 Acked 9413 Write 1 Done Done New false 8415 Acked 9414 Read 5 Done Done New false 8416 Acked 9415 Read 6 Done Done New false 8417 Acked 9416 Read 8 Done Done New false 8418 Acked 9417 Write 8 Done Done New false 8419 Acked 9418 Read 3 Done Done New false 8420 Acked 9419 Read 1 Done Done New false 8421 Acked 9420 Read 10 Done Done New false 8422 Acked 9421 Read 9 Done Done New false 8423 Acked 9422 Read 6 Done Done New false 8424 Acked 9423 Write 8 Done Done New false 8425 Acked 9424 Read 7 Done Done New false 8426 Acked 9425 Read 8 Done Done New false 8427 Acked 9426 Read 6 Done Done New false 8428 Acked 9427 Read 4 Done Done New false 8429 Acked 9428 Read 9 Done Done New false 8430 Acked 9429 Write 8 Done Done New false 8431 Acked 9430 Write 4 Done Done New false 8432 Acked 9431 Read 2 Done Done New false 8433 Acked 9432 Write 8 Done Done New false 8434 Acked 9433 Flush 0 Done Done New false 8435 Acked 9434 Read 1 Done Done New false 8436 Acked 9435 Read 2 Done Done New false 8437 Acked 9436 Read 9 Done Done New false 8438 Acked 9437 Flush 0 Done Done New false 8439 Acked 9438 Flush 0 Done Done New false 8440 Acked 9439 Write 6 Done Done New false 8441 Acked 9440 Write 10 Done Done New false 8442 Acked 9441 Write 10 Done Done New false 8443 Acked 9442 Write 8 Done Done New false 8444 Acked 9443 Read 1 Done Done New false 8445 Acked 9444 Flush 0 Done Done New false 8446 Acked 9445 Write 2 Done Done New false 8447 Acked 9446 Write 9 Done Done New false 8448 Acked 9447 Write 7 Done Done New false 8449 Acked 9448 Write 2 Done Done New false 8450 Acked 9449 Flush 0 Done Done New false 8451 Acked 9450 Write 7 Done Done New false 8452 Acked 9451 Read 10 Done Done New false 8453 Acked 9452 Read 10 Done Done New false 8454 Acked 9453 Write 4 Done Done New false 8455 Acked 9454 Read 1 Done Done New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 302 302 Sent 0 0 0 0 Done 302 302 0 604 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9449 9449 9152 Downstairs last five completed: 9152 9151 9150 9149 9148 Upstairs last five completed: 8455 8454 8453 8452 8451 CLIENT: Up:0 ds:302 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:03.404240157Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:03.404361958Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:03.40442439Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:03.404487956Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:03.404508592Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:03.404531517Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:03.404798699Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:04.405676144Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:04.40574477Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:04.405773131Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:04.405809809Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:04.405831501Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 303 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:04.405854132Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:04.406093699Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:05.406986692Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:05.407056756Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:05.407085256Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:05.407111514Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:05.407131796Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 303 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:05.407197075Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:05.407436704Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:06.408394203Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:06.408478569Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:06.408535335Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:06.408570634Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:06.408617625Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:06.408655556Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 303 jobs since flush 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:06.408691978Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:06.40879536Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1213) [8830][2] reports Running ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 8154 Acked 9153 Write 5 Done Done New true 8155 Acked 9154 Write 4 Done Done New true 8156 Acked 9155 Flush 0 Done Done New true 8157 Acked 9156 Read 6 Done Done New true 8158 Acked 9157 Read 6 Done Done New true 8159 Acked 9158 Read 10 Done Done New true 8160 Acked 9159 Write 9 Done Done New true 8161 Acked 9160 Write 4 Done Done New true 8162 Acked 9161 Read 3 Done Done New true 8163 Acked 9162 Read 7 Done Done New true 8164 Acked 9163 Read 4 Done Done New true 8165 Acked 9164 Flush 0 Done Done New true 8166 Acked 9165 Write 10 Done Done New true 8167 Acked 9166 Write 7 Done Done New true 8168 Acked 9167 Flush 0 Done Done New true 8169 Acked 9168 Write 10 Done Done New true 8170 Acked 9169 Write 1 Done Done New true 8171 Acked 9170 Read 9 Done Done New true 8172 Acked 9171 Write 6 Done Done New true 8173 Acked 9172 Read 3 Done Done New true 8174 Acked 9173 Write 3 Done Done New true 8175 Acked 9174 Write 1 Done Done New true 8176 Acked 9175 Flush 0 Done Done New true 8177 Acked 9176 Read 7 Done Done New true 8178 Acked 9177 Write 4 Done Done New true 8179 Acked 9178 Write 1 Done Done New true 8180 Acked 9179 Read 4 Done Done New true 8181 Acked 9180 Write 1 Done Done New true 8182 Acked 9181 Read 3 Done Done New true 8183 Acked 9182 Read 4 Done Done New true 8184 Acked 9183 Write 4 Done Done New true 8185 Acked 9184 Read 1 Done Done New true 8186 Acked 9185 Write 8 Done Done New true 8187 Acked 9186 Write 3 Done Done New true 8188 Acked 9187 Write 10 Done Done New true 8189 Acked 9188 Read 2 Done Done New true 8190 Acked 9189 Read 2 Done Done New true 8191 Acked 9190 Flush 0 Done Done New true 8192 Acked 9191 Read 8 Done Done New true 8193 Acked 9192 Write 8 Done Done New true 8194 Acked 9193 Write 3 Done Done New true 8195 Acked 9194 Read 10 Done Done New true 8196 Acked 9195 Write 3 Done Done New true 8197 Acked 9196 Read 4 Done Done New true 8198 Acked 9197 Read 2 Done Done New true 8199 Acked 9198 Read 2 Done Done New true 8200 Acked 9199 Read 10 Done Done New true 8201 Acked 9200 Read 5 Done Done New true 8202 Acked 9201 Write 3 Done Done New true 8203 Acked 9202 Read 2 Done Done New true 8204 Acked 9203 Read 8 Done Done New true 8205 Acked 9204 Write 7 Done Done New true 8206 Acked 9205 Write 1 Done Done New true 8207 Acked 9206 Read 2 Done Done New true 8208 Acked 9207 Read 6 Done Done New true 8209 Acked 9208 Flush 0 Done Done New true 8210 Acked 9209 Read 7 Done Done New true 8211 Acked 9210 Write 7 Done Done New true 8212 Acked 9211 Write 10 Done Done New true 8213 Acked 9212 Read 4 Done Done New true 8214 Acked 9213 Write 2 Done Done New true 8215 Acked 9214 Read 9 Done Done New true 8216 Acked 9215 Read 7 Done Done New true 8217 Acked 9216 Write 10 Done Done New true 8218 Acked 9217 Write 1 Done Done New true 8219 Acked 9218 Flush 0 Done Done New true 8220 Acked 9219 Read 2 Done Done New true 8221 Acked 9220 Write 5 Done Done New true 8222 Acked 9221 Read 8 Done Done New true 8223 Acked 9222 Write 4 Done Done New true 8224 Acked 9223 Read 7 Done Done New true 8225 Acked 9224 Write 1 Done Done New true 8226 Acked 9225 Write 7 Done Done New true 8227 Acked 9226 Read 3 Done Done New true 8228 Acked 9227 Flush 0 Done Done New true 8229 Acked 9228 Write 6 Done Done New true 8230 Acked 9229 Read 7 Done Done New true 8231 Acked 9230 Write 5 Done Done New true 8232 Acked 9231 Write 10 Done Done New true 8233 Acked 9232 Write 5 Done Done New true 8234 Acked 9233 Read 7 Done Done New true 8235 Acked 9234 Write 8 Done Done New true 8236 Acked 9235 Read 6 Done Done New true 8237 Acked 9236 Write 10 Done Done New true 8238 Acked 9237 Read 4 Done Done New true 8239 Acked 9238 Read 5 Done Done New true 8240 Acked 9239 Flush 0 Done Done New true 8241 Acked 9240 Write 4 Done Done New true 8242 Acked 9241 Read 8 Done Done New true 8243 Acked 9242 Flush 0 Done Done New true 8244 Acked 9243 Read 10 Done Done New true 8245 Acked 9244 Read 5 Done Done New true 8246 Acked 9245 Flush 0 Done Done New true 8247 Acked 9246 Read 5 Done Done New true 8248 Acked 9247 Write 9 Done Done New true 8249 Acked 9248 Write 5 Done Done New true 8250 Acked 9249 Write 10 Done Done New true 8251 Acked 9250 Read 5 Done Done New true 8252 Acked 9251 Write 3 Done Done New true 8253 Acked 9252 Write 2 Done Done New true 8254 Acked 9253 Flush 0 Done Done New true 8255 Acked 9254 Read 8 Done Done New true 8256 Acked 9255 Read 2 Done Done New true 8257 Acked 9256 Read 2 Done Done New true 8258 Acked 9257 Write 5 Done Done New true 8259 Acked 9258 Write 2 Done Done New true 8260 Acked 9259 Read 5 Done Done New true 8261 Acked 9260 Flush 0 Done Done New true 8262 Acked 9261 Read 7 Done Done New true 8263 Acked 9262 Write 7 Done Done New true 8264 Acked 9263 Flush 0 Done Done New true 8265 Acked 9264 Read 1 Done Done New true 8266 Acked 9265 Write 10 Done Done New true 8267 Acked 9266 Write 2 Done Done New true 8268 Acked 9267 Write 3 Done Done New true 8269 Acked 9268 Read 7 Done Done New true 8270 Acked 9269 Read 8 Done Done New true 8271 Acked 9270 Read 10 Done Done New true 8272 Acked 9271 Read 6 Done Done New true 8273 Acked 9272 Read 3 Done Done New true 8274 Acked 9273 Read 7 Done Done New true 8275 Acked 9274 Read 2 Done Done New true 8276 Acked 9275 Flush 0 Done Done New true 8277 Acked 9276 Flush 0 Done Done New true 8278 Acked 9277 Read 1 Done Done New true 8279 Acked 9278 Flush 0 Done Done New true 8280 Acked 9279 Read 1 Done Done New true 8281 Acked 9280 Write 8 Done Done New true 8282 Acked 9281 Read 3 Done Done New true 8283 Acked 9282 Read 1 Done Done New true 8284 Acked 9283 Read 3 Done Done New true 8285 Acked 9284 Read 4 Done Done New true 8286 Acked 9285 Write 7 Done Done New true 8287 Acked 9286 Read 7 Done Done New true 8288 Acked 9287 Read 8 Done Done New true 8289 Acked 9288 Read 9 Done Done New true 8290 Acked 9289 Write 9 Done Done New true 8291 Acked 9290 Write 5 Done Done New true 8292 Acked 9291 Write 5 Done Done New true 8293 Acked 9292 Read 8 Done Done New true 8294 Acked 9293 Write 2 Done Done New true 8295 Acked 9294 Write 10 Done Done New true 8296 Acked 9295 Read 3 Done Done New true 8297 Acked 9296 Flush 0 Done Done New true 8298 Acked 9297 Read 6 Done Done New true 8299 Acked 9298 Write 6 Done Done New true 8300 Acked 9299 Read 8 Done Done New true 8301 Acked 9300 Write 5 Done Done New true 8302 Acked 9301 Read 3 Done Done New true 8303 Acked 9302 Write 9 Done Done New true 8304 Acked 9303 Write 7 Done Done New true 8305 Acked 9304 Write 10 Done Done New true 8306 Acked 9305 Write 9 Done Done New true 8307 Acked 9306 Write 2 Done Done New true 8308 Acked 9307 Write 2 Done Done New true 8309 Acked 9308 Read 3 Done Done New true 8310 Acked 9309 Write 3 Done Done New true 8311 Acked 9310 Write 2 Done Done New true 8312 Acked 9311 Read 9 Done Done New true 8313 Acked 9312 Flush 0 Done Done New true 8314 Acked 9313 Read 4 Done Done New true 8315 Acked 9314 Read 2 Done Done New true 8316 Acked 9315 Read 2 Done Done New true 8317 Acked 9316 Write 4 Done Done New true 8318 Acked 9317 Read 9 Done Done New true 8319 Acked 9318 Read 10 Done Done New true 8320 Acked 9319 Read 2 Done Done New true 8321 Acked 9320 Read 8 Done Done New true 8322 Acked 9321 Flush 0 Done Done New true 8323 Acked 9322 Write 6 Done Done New true 8324 Acked 9323 Write 6 Done Done New true 8325 Acked 9324 Read 9 Done Done New true 8326 Acked 9325 Read 3 Done Done New true 8327 Acked 9326 Flush 0 Done Done New true 8328 Acked 9327 Write 8 Done Done New true 8329 Acked 9328 Write 7 Done Done New true 8330 Acked 9329 Flush 0 Done Done New true 8331 Acked 9330 Read 3 Done Done New true 8332 Acked 9331 Write 7 Done Done New true 8333 Acked 9332 Read 5 Done Done New true 8334 Acked 9333 Write 10 Done Done New true 8335 Acked 9334 Read 9 Done Done New true 8336 Acked 9335 Write 10 Done Done New true 8337 Acked 9336 Read 1 Done Done New true 8338 Acked 9337 Read 1 Done Done New true 8339 Acked 9338 Read 3 Done Done New true 8340 Acked 9339 Read 4 Done Done New true 8341 Acked 9340 Read 6 Done Done New true 8342 Acked 9341 Read 4 Done Done New true 8343 Acked 9342 Read 2 Done Done New true 8344 Acked 9343 Write 2 Done Done New true 8345 Acked 9344 Write 2 Done Done New true 8346 Acked 9345 Write 6 Done Done New true 8347 Acked 9346 Write 5 Done Done New true 8348 Acked 9347 Read 9 Done Done New true 8349 Acked 9348 Write 10 Done Done New true 8350 Acked 9349 Write 7 Done Done New true 8351 Acked 9350 Write 9 Done Done New true 8352 Acked 9351 Write 1 Done Done New true 8353 Acked 9352 Write 6 Done Done New true 8354 Acked 9353 Write 3 Done Done New true 8355 Acked 9354 Read 7 Done Done New true 8356 Acked 9355 Read 2 Done Done New true 8357 Acked 9356 Write 4 Done Done New true 8358 Acked 9357 Read 8 Done Done New true 8359 Acked 9358 Read 4 Done Done New true 8360 Acked 9359 Read 8 Done Done New true 8361 Acked 9360 Read 5 Done Done New true 8362 Acked 9361 Read 10 Done Done New true 8363 Acked 9362 Read 1 Done Done New true 8364 Acked 9363 Flush 0 Done Done New true 8365 Acked 9364 Flush 0 Done Done New true 8366 Acked 9365 Read 1 Done Done New true 8367 Acked 9366 Read 4 Done Done New true 8368 Acked 9367 Read 9 Done Done New true 8369 Acked 9368 Read 10 Done Done New true 8370 Acked 9369 Read 10 Done Done New true 8371 Acked 9370 Read 7 Done Done New true 8372 Acked 9371 Read 5 Done Done New true 8373 Acked 9372 Read 4 Done Done New true 8374 Acked 9373 Write 3 Done Done New true 8375 Acked 9374 Read 7 Done Done New true 8376 Acked 9375 Write 3 Done Done New true 8377 Acked 9376 Read 7 Done Done New true 8378 Acked 9377 Read 3 Done Done New true 8379 Acked 9378 Read 1 Done Done New true 8380 Acked 9379 Write 3 Done Done New true 8381 Acked 9380 Read 10 Done Done New true 8382 Acked 9381 Read 1 Done Done New true 8383 Acked 9382 Write 7 Done Done New true 8384 Acked 9383 Write 5 Done Done New true 8385 Acked 9384 Write 7 Done Done New true 8386 Acked 9385 Flush 0 Done Done New true 8387 Acked 9386 Write 8 Done Done New true 8388 Acked 9387 Write 6 Done Done New true 8389 Acked 9388 Write 7 Done Done New true 8390 Acked 9389 Read 1 Done Done New true 8391 Acked 9390 Flush 0 Done Done New true 8392 Acked 9391 Read 5 Done Done New true 8393 Acked 9392 Flush 0 Done Done New true 8394 Acked 9393 Read 8 Done Done New true 8395 Acked 9394 Write 9 Done Done New true 8396 Acked 9395 Flush 0 Done Done New true 8397 Acked 9396 Read 3 Done Done New true 8398 Acked 9397 Read 8 Done Done New true 8399 Acked 9398 Write 5 Done Done New true 8400 Acked 9399 Write 7 Done Done New true 8401 Acked 9400 Write 3 Done Done New true 8402 Acked 9401 Write 1 Done Done New true 8403 Acked 9402 Write 4 Done Done New true 8404 Acked 9403 Write 6 Done Done New true 8405 Acked 9404 Read 5 Done Done New true 8406 Acked 9405 Read 4 Done Done New true 8407 Acked 9406 Write 9 Done Done New true 8408 Acked 9407 Write 2 Done Done New true 8409 Acked 9408 Read 6 Done Done New true 8410 Acked 9409 Write 8 Done Done New true 8411 Acked 9410 Write 8 Done Done New true 8412 Acked 9411 Write 3 Done Done New true 8413 Acked 9412 Read 10 Done Done New true 8414 Acked 9413 Write 1 Done Done New true 8415 Acked 9414 Read 5 Done Done New true 8416 Acked 9415 Read 6 Done Done New true 8417 Acked 9416 Read 8 Done Done New true 8418 Acked 9417 Write 8 Done Done New true 8419 Acked 9418 Read 3 Done Done New true 8420 Acked 9419 Read 1 Done Done New true 8421 Acked 9420 Read 10 Done Done New true 8422 Acked 9421 Read 9 Done Done New true 8423 Acked 9422 Read 6 Done Done New true 8424 Acked 9423 Write 8 Done Done New true 8425 Acked 9424 Read 7 Done Done New true 8426 Acked 9425 Read 8 Done Done New true 8427 Acked 9426 Read 6 Done Done New true 8428 Acked 9427 Read 4 Done Done New true 8429 Acked 9428 Read 9 Done Done New true 8430 Acked 9429 Write 8 Done Done New true 8431 Acked 9430 Write 4 Done Done New true 8432 Acked 9431 Read 2 Done Done New true 8433 Acked 9432 Write 8 Done Done New true 8434 Acked 9433 Flush 0 Done Done New true 8435 Acked 9434 Read 1 Done Done New true 8436 Acked 9435 Read 2 Done Done New true 8437 Acked 9436 Read 9 Done Done New true 8438 Acked 9437 Flush 0 Done Done New true 8439 Acked 9438 Flush 0 Done Done New true 8440 Acked 9439 Write 6 Done Done New true 8441 Acked 9440 Write 10 Done Done New true 8442 Acked 9441 Write 10 Done Done New true 8443 Acked 9442 Write 8 Done Done New true 8444 Acked 9443 Read 1 Done Done New true 8445 Acked 9444 Flush 0 Done Done New true 8446 Acked 9445 Write 2 Done Done New true 8447 Acked 9446 Write 9 Done Done New true 8448 Acked 9447 Write 7 Done Done New true 8449 Acked 9448 Write 2 Done Done New true 8450 Acked 9449 Flush 0 Done Done New true 8451 Acked 9450 Write 7 Done Done New true 8452 Acked 9451 Read 10 Done Done New true 8453 Acked 9452 Read 10 Done Done New true 8454 Acked 9453 Write 4 Done Done New true 8455 Acked 9454 Read 1 Done Done New true 8456 Acked 9455 Flush 0 Done Done New true STATES DS:0 DS:1 DS:2 TOTAL New 0 0 303 303 Sent 0 0 0 0 Done 303 303 0 606 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9455 9455 9152 Downstairs last five completed: 9152 9151 9150 9149 9148 Upstairs last five completed: 8456 8455 8454 8453 8452 CLIENT: Up:0 ds:303 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.410071046Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.410215771Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412251867Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412294878Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412319645Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 9152","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412594083Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412624204Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412647587Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412669919Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:07.412695865Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 9455 9454 9453 9452 9451 Upstairs last five completed: 8456 8455 8454 8453 8452 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(2) stop 2 [8830] Got stop action so:false kr:false Jan 02 22:36:11.030 INFO request completed, latency_us: 471, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2e10deaf-b52b-435c-b949-9b8606d8d822, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:36:11.031 INFO accepted connection, remote_addr: 127.0.0.1:43580, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:11.031710381Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:11.031836462Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:36:11.031 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:11.031898194Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} GET, req_id: 16a1072a-b509-4594-998b-9afd9f442a70, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:11.031947004Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:11.031995003Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:11.032040407Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:12.033514472Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:12.033628223Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:12.03365697Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:12.033681888Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:12.033708555Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:12.033740747Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:12.033764661Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:12.033787542Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:13.034814671Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:13.034924352Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:13.034984604Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:13.035050839Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:13.035076141Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:13.03509967Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:13.035123864Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:14.036331436Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:14.036397305Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:14.036424571Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:14.036450728Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:14.03648096Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 0 jobs since flush 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:14.0365047Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:14.036529035Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:36:15.034 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/2, method: GET, req_id: 097e1d0b-25e9-4016-bcb4-e8a8ef0b0304, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Read block 1470 len 8192 002/300 Read block 3300 len 36864 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:15.037078082Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:15.037142582Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:15.037170627Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:15.037197109Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:15.03721744Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 2 jobs since flush 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:15.037337686Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:15.037478857Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 003/300 Read block 4537 len 16384 004/300 Flush 005/300 Write block 1267 len 16384 data: 6 5 5 7 006/300 Read block 955 len 8192 007/300 Write block 1099 len 20480 data: 6 6 6 7 8 008/300 Write block 946 len 40960 data: 8 5 7 7 7 8 6 7 6 6 009/300 Read block 4811 len 36864 010/300 Write block 856 len 40960 data: 4 4 6 6 7 6 6 5 7 6 011/300 Write block 747 len 32768 data: 4 2 2 2 2 3 3 4 012/300 Write block 3131 len 12288 data: 4 6 7 013/300 Write block 1220 len 12288 data: 8 8 7 014/300 Write block 650 len 16384 data: 4 4 5 5 015/300 Write block 754 len 12288 data: 5 4 6 016/300 Write block 2613 len 16384 data: 7 7 8 8 017/300 Read block 1871 len 8192 018/300 Read block 4978 len 8192 019/300 Read block 870 len 8192 020/300 Read block 1308 len 8192 021/300 Read block 3502 len 32768 022/300 Read block 1415 len 20480 023/300 Write block 3461 len 4096 data: 4 024/300 Read block 1562 len 28672 025/300 Read block 1173 len 40960 026/300 Read block 2421 len 16384 027/300 Read block 2896 len 20480 028/300 Read block 4399 len 36864 029/300 Write block 2237 len 28672 data: 6 6 6 7 4 4 4 030/300 Read block 1526 len 16384 031/300 Write block 4203 len 24576 data: 5 6 7 6 7 6 032/300 Read block 3643 len 40960 033/300 Read block 1919 len 16384 034/300 Write block 2314 len 12288 data: 6 4 4 035/300 Write block 4560 len 32768 data: 3 4 5 6 6 7 8 8 036/300 Read block 4537 len 16384 037/300 Read block 3217 len 16384 038/300 Read block 1763 len 24576 039/300 Read block 3107 len 28672 040/300 Read block 361 len 24576 041/300 Write block 3916 len 12288 data: 3 4 4 042/300 Write block 1381 len 12288 data: 3 3 3 043/300 Write block 400 len 16384 data: 6 6 6 8 044/300 Write block 628 len 20480 data: 7 6 5 5 5 045/300 Read block 1353 len 32768 046/300 Read block 4358 len 8192 047/300 Read block 4760 len 16384 048/300 Write block 3880 len 4096 data: 7 049/300 Read block 3080 len 12288 050/300 Write block 1498 len 32768 data: 6 7 9 7 6 8 9 8 051/300 Write block 2741 len 24576 data: 5 5 4 5 5 6 052/300 Read block 2664 len 28672 053/300 Read block 2139 len 20480 054/300 Write block 2766 len 8192 data: 8 8 055/300 Read block 4453 len 28672 056/300 Read block 977 len 16384 057/300 Write block 4207 len 24576 data: 8 7 7 5 5 5 058/300 Read block 2920 len 24576 059/300 Write block 3296 len 40960 data: 4 4 6 7 8 8 7 5 7 7 060/300 Read block 2314 len 36864 061/300 Read block 4687 len 36864 062/300 Read block 3951 len 12288 063/300 Write block 1858 len 36864 data: 2 2 2 3 3 3 4 4 5 064/300 Read block 3541 len 32768 065/300 Write block 3517 len 20480 data: 6 6 5 4 3 066/300 Read block 2022 len 8192 067/300 Write block 2053 len 16384 data: 8 7 7 8 068/300 Flush 069/300 Write block 3510 len 32768 data: 4 5 5 8 6 7 6 7 070/300 Read block 3096 len 32768 071/300 Read block 1033 len 40960 072/300 Read block 4524 len 8192 073/300 Write block 671 len 32768 data: 6 5 4 5 6 6 5 6 074/300 Read block 3980 len 24576 075/300 Write block 4757 len 8192 data: 8 6 076/300 Write block 4090 len 20480 data: 10 11 11 9 9 077/300 Read block 1195 len 20480 078/300 Write block 3219 len 4096 data: 6 079/300 Read block 133 len 12288 080/300 Write block 2158 len 20480 data: 9 6 7 6 6 081/300 Read block 874 len 32768 082/300 Write block 1819 len 20480 data: 4 4 4 4 5 083/300 Read block 3402 len 32768 084/300 Write block 1414 len 40960 data: 4 5 6 5 5 6 6 7 7 6 085/300 Write block 1153 len 4096 data: 4 086/300 Write block 72 len 28672 data: 3 2 2 4 4 5 6 087/300 Write block 4037 len 36864 data: 4 4 4 5 5 6 8 7 9 088/300 Read block 3119 len 28672 089/300 Read block 2741 len 32768 090/300 Read block 313 len 20480 091/300 Flush 092/300 Write block 2786 len 20480 data: 7 8 6 6 5 093/300 Read block 1758 len 32768 094/300 Read block 3241 len 32768 095/300 Flush 096/300 Read block 1601 len 20480 097/300 Write block 3000 len 12288 data: 4 4 3 098/300 Read block 1330 len 28672 099/300 Read block 2415 len 36864 100/300 Read block 4957 len 4096 101/300 Write block 1928 len 4096 data: 5 102/300 Read block 604 len 16384 103/300 Read block 2520 len 16384 104/300 Write block 4904 len 20480 data: 6 5 5 6 6 105/300 Read block 1894 len 32768 106/300 Write block 795 len 36864 data: 5 5 5 3 4 4 6 6 7 107/300 Read block 1345 len 40960 108/300 Write block 1333 len 32768 data: 8 7 5 4 4 4 4 5 109/300 Write block 4097 len 36864 data: 6 6 6 8 8 9 5 5 4 110/300 Write block 600 len 4096 data: 7 111/300 Read block 667 len 28672 112/300 Read block 1304 len 12288 113/300 Read block 2349 len 8192 114/300 Write block 2219 len 32768 data: 5 4 5 4 6 6 7 9 115/300 Write block 4342 len 40960 data: 6 7 7 8 6 4 6 5 6 6 116/300 Flush 117/300 Write block 4787 len 12288 data: 6 5 4 118/300 Write block 4275 len 8192 data: 4 5 119/300 Read block 2945 len 4096 120/300 Read block 4023 len 28672 121/300 Read block 4477 len 36864 122/300 Read block 4849 len 32768 123/300 Read block 2432 len 20480 124/300 Write block 276 len 16384 data: 6 5 5 5 125/300 Write block 3829 len 36864 data: 5 5 4 4 5 5 4 4 5 126/300 Write block 125 len 40960 data: 6 5 6 6 8 8 8 7 8 8 127/300 Write block 848 len 28672 data: 7 7 8 8 7 7 6 128/300 Flush 129/300 Write block 1 len 12288 data: 3 4 4 130/300 Read block 2367 len 8192 131/300 Flush 132/300 Read block 167 len 36864 133/300 Write block 3005 len 8192 data: 5 5 134/300 Write block 1527 len 36864 data: 8 3 4 4 4 4 6 5 5 135/300 Read block 2682 len 32768 136/300 Read block 4194 len 20480 137/300 Read block 1296 len 40960 138/300 Write block 3898 len 36864 data: 6 5 3 3 3 3 3 3 5 139/300 Read block 1064 len 28672 140/300 Write block 4760 len 24576 data: 4 5 5 4 4 5 141/300 Read block 1527 len 28672 142/300 Read block 2610 len 32768 143/300 Read block 3565 len 24576 144/300 Read block 4943 len 20480 145/300 Write block 4707 len 28672 data: 3 3 4 5 5 4 5 146/300 Read block 2435 len 40960 147/300 Write block 148 len 4096 data: 9 148/300 Write block 2974 len 36864 data: 6 6 5 5 5 5 4 4 3 149/300 Read block 3150 len 8192 150/300 Read block 3990 len 16384 151/300 Read block 2159 len 12288 152/300 Write block 3831 len 40960 data: 5 5 6 6 5 5 6 5 6 6 153/300 Write block 4975 len 20480 data: 6 7 4 5 8 154/300 Write block 2682 len 28672 data: 6 5 5 7 7 7 9 155/300 Flush 156/300 Write block 3455 len 32768 data: 10 9 8 6 5 4 5 3 157/300 Read block 1939 len 4096 158/300 Write block 2290 len 32768 data: 4 3 2 3 3 5 6 6 159/300 Write block 3663 len 4096 data: 7 160/300 Read block 1833 len 36864 161/300 Read block 261 len 32768 162/300 Read block 1334 len 36864 163/300 Write block 4162 len 4096 data: 4 164/300 Write block 4291 len 32768 data: 5 5 5 7 8 6 5 4 165/300 Write block 3281 len 40960 data: 3 3 4 4 4 4 3 4 3 3 166/300 Write block 354 len 32768 data: 6 5 4 4 4 3 4 5 167/300 Write block 2906 len 32768 data: 8 6 6 5 4 5 6 8 168/300 Write block 1612 len 12288 data: 7 6 4 169/300 Read block 1716 len 8192 170/300 Read block 4457 len 28672 171/300 Read block 351 len 20480 172/300 Write block 2089 len 32768 data: 2 2 2 2 3 3 3 4 173/300 Write block 33 len 40960 data: 4 5 6 7 6 6 7 6 6 6 174/300 Read block 2911 len 28672 175/300 Read block 125 len 16384 176/300 Read block 453 len 24576 177/300 Read block 253 len 24576 178/300 Write block 4267 len 16384 data: 8 12 14 12 179/300 Flush 180/300 Write block 552 len 4096 data: 4 181/300 Write block 2966 len 36864 data: 6 8 8 9 8 6 6 6 7 182/300 Write block 4325 len 36864 data: 10 11 10 9 8 6 5 5 7 183/300 Read block 2278 len 12288 184/300 Write block 4368 len 8192 data: 5 5 185/300 Write block 4033 len 24576 data: 4 3 3 3 5 5 186/300 Read block 1320 len 12288 187/300 Read block 4246 len 20480 188/300 Write block 2590 len 12288 data: 6 6 4 189/300 Write block 4900 len 24576 data: 5 5 6 6 7 6 190/300 Write block 2202 len 24576 data: 6 5 5 5 4 5 191/300 Flush 192/300 Read block 1628 len 24576 193/300 Write block 1947 len 12288 data: 7 7 6 194/300 Read block 2432 len 24576 195/300 Write block 4288 len 4096 data: 5 196/300 Flush 197/300 Read block 4422 len 20480 198/300 Read block 2684 len 12288 199/300 Write block 4026 len 32768 data: 5 5 3 3 3 3 3 5 200/300 Write block 4776 len 36864 data: 7 6 7 7 4 3 4 4 3 201/300 Read block 828 len 36864 202/300 Read block 2221 len 12288 203/300 Read block 1770 len 40960 204/300 Flush 205/300 Read block 3407 len 12288 206/300 Write block 3511 len 40960 data: 6 6 9 7 8 7 8 7 6 5 207/300 Write block 2468 len 36864 data: 8 7 4 6 4 5 5 6 8 208/300 Flush 209/300 Read block 2677 len 4096 210/300 Write block 3723 len 28672 data: 7 6 7 7 7 7 8 211/300 Read block 3952 len 8192 212/300 Write block 4526 len 12288 data: 3 3 4 213/300 Read block 4627 len 36864 214/300 Read block 249 len 8192 215/300 Read block 1839 len 16384 216/300 Write block 4917 len 4096 data: 3 217/300 Read block 3162 len 12288 218/300 Write block 2631 len 36864 data: 5 7 7 7 10 10 10 9 9 219/300 Write block 3235 len 4096 data: 6 220/300 Write block 2091 len 8192 data: 3 3 221/300 Read block 2484 len 20480 222/300 Read block 2582 len 32768 223/300 Flush 224/300 Read block 955 len 32768 225/300 Flush 226/300 Write block 4145 len 16384 data: 8 7 6 8 227/300 Read block 2449 len 8192 228/300 Read block 4094 len 4096 229/300 Write block 1314 len 8192 data: 3 3 230/300 Read block 4830 len 40960 231/300 Write block 4094 len 4096 data: 10 232/300 Read block 3328 len 16384 233/300 Read block 1085 len 32768 234/300 Write block 3606 len 12288 data: 9 8 9 235/300 Write block 3683 len 32768 data: 5 5 6 4 4 3 4 4 236/300 Read block 1361 len 20480 237/300 Write block 3944 len 28672 data: 5 4 3 4 3 3 3 238/300 Write block 2526 len 36864 data: 4 3 3 6 6 6 8 8 7 239/300 Write block 3325 len 16384 data: 4 5 3 3 240/300 Write block 1909 len 20480 data: 3 4 6 7 6 241/300 Read block 4781 len 4096 242/300 Read block 4823 len 24576 243/300 Read block 1809 len 8192 244/300 Write block 82 len 8192 data: 5 6 245/300 Read block 2610 len 28672 246/300 Read block 2093 len 20480 247/300 Read block 1342 len 32768 248/300 Read block 1197 len 20480 249/300 Read block 315 len 28672 250/300 Write block 2054 len 36864 data: 8 8 9 8 6 6 5 4 4 251/300 Write block 513 len 28672 data: 6 7 8 10 7 8 6 252/300 Read block 3515 len 36864 253/300 Write block 1739 len 16384 data: 6 7 4 5 254/300 Read block 3043 len 36864 255/300 Write block 1933 len 4096 data: 3 256/300 Write block 3059 len 12288 data: 10 12 11 257/300 Write block 3494 len 28672 data: 5 4 5 5 4 4 5 258/300 Read block 2453 len 24576 259/300 Read block 2662 len 28672 260/300 Read block 1891 len 16384 261/300 Read block 4726 len 28672 262/300 Write block 4473 len 40960 data: 4 6 6 5 6 5 5 3 3 3 263/300 Flush 264/300 Read block 1769 len 16384 265/300 Write block 1373 len 40960 data: 3 3 4 4 4 4 6 4 4 4 266/300 Write block 4010 len 12288 data: 13 11 10 267/300 Read block 483 len 12288 268/300 Read block 1699 len 4096 269/300 Read block 966 len 40960 270/300 Write block 429 len 4096 data: 9 271/300 Write block 1952 len 8192 data: 3 4 272/300 Read block 2187 len 32768 273/300 Read block 4588 len 20480 274/300 Write block 753 len 12288 data: 4 6 5 275/300 Read block 1042 len 28672 276/300 Write block 3179 len 32768 data: 7 7 6 7 7 7 5 6 277/300 Write block 2486 len 40960 data: 7 7 8 7 7 7 6 6 4 3 278/300 Read block 2417 len 36864 279/300 Read block 1534 len 20480 280/300 Write block 4329 len 28672 data: 9 7 6 6 8 7 9 281/300 Read block 976 len 12288 282/300 Read block 3692 len 20480 283/300 Flush 284/300 Read block 2181 len 36864 285/300 Flush 286/300 Write block 894 len 36864 data: 6 6 6 7 8 7 7 4 4 287/300 Read block 2074 len 16384 288/300 Read block 2530 len 28672 289/300 Write block 2551 len 4096 data: 6 290/300 Write block 4559 len 8192 data: 3 4 291/300 Write block 3502 len 36864 data: 5 5 6 6 5 4 4 4 5 292/300 Write block 1348 len 4096 data: 9 293/300 Read block 4159 len 16384 294/300 Write block 431 len 20480 data: 9 8 7 8 8 295/300 Flush 296/300 Read block 4311 len 40960 297/300 Read block 3664 len 16384 298/300 Write block 591 len 8192 data: 3 3 299/300 Write block 4765 len 4096 data: 6 300/300 Read block 4167 len 40960 Main task has work to do, go find it got dsc Start(2) start 2 Jan 02 22:36:15.648 INFO request completed, latency_us: 558, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7cd0c2f6-1222-4c19-bffe-1abc349e8ffa, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 2, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Active Active Offline GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 8457 Acked 9456 Read 2 Done Done New true 8458 Acked 9457 Read 9 Done Done New true 8459 Acked 9458 Read 4 Done Done New false 8460 Acked 9459 Flush 0 Done Done New false 8461 Acked 9460 Write 4 Done Done New false 8462 Acked 9461 Read 2 Done Done New false 8463 Acked 9462 Write 5 Done Done New false 8464 Acked 9463 Write 10 Done Done New false 8465 Acked 9464 Read 9 Done Done New false 8466 Acked 9465 Write 10 Done Done New false 8467 Acked 9466 Write 8 Done Done New false 8468 Acked 9467 Write 3 Done Done New false 8469 Acked 9468 Write 3 Done Done New false 8470 Acked 9469 Write 4 Done Done New false 8471 Acked 9470 Write 3 Done Done New false 8472 Acked 9471 Write 4 Done Done New false 8473 Acked 9472 Read 2 Done Done New false 8474 Acked 9473 Read 2 Done Done New false 8475 Acked 9474 Read 2 Done Done New false 8476 Acked 9475 Read 2 Done Done New false 8477 Acked 9476 Read 8 Done Done New false 8478 Acked 9477 Read 5 Done Done New false 8479 Acked 9478 Write 1 Done Done New false 8480 Acked 9479 Read 7 Done Done New false 8481 Acked 9480 Read 10 Done Done New false 8482 Acked 9481 Read 4 Done Done New false 8483 Acked 9482 Read 5 Done Done New false 8484 Acked 9483 Read 9 Done Done New false 8485 Acked 9484 Write 7 Done Done New false 8486 Acked 9485 Read 4 Done Done New false 8487 Acked 9486 Write 6 Done Done New false 8488 Acked 9487 Read 10 Done Done New false 8489 Acked 9488 Read 4 Done Done New false 8490 Acked 9489 Write 3 Done Done New false 8491 Acked 9490 Write 8 Done Done New false 8492 Acked 9491 Read 4 Done Done New false 8493 Acked 9492 Read 4 Done Done New false 8494 Acked 9493 Read 6 Done Done New false 8495 Acked 9494 Read 7 Done Done New false 8496 Acked 9495 Read 6 Done Done New false 8497 Acked 9496 Write 3 Done Done New false 8498 Acked 9497 Write 3 Done Done New false 8499 Acked 9498 Write 4 Done Done New false 8500 Acked 9499 Write 5 Done Done New false 8501 Acked 9500 Read 8 Done Done New false 8502 Acked 9501 Read 2 Done Done New false 8503 Acked 9502 Read 4 Done Done New false 8504 Acked 9503 Write 1 Done Done New false 8505 Acked 9504 Read 3 Done Done New false 8506 Acked 9505 Write 8 Done Done New false 8507 Acked 9506 Write 6 Done Done New false 8508 Acked 9507 Read 7 Done Done New false 8509 Acked 9508 Read 5 Done Done New false 8510 Acked 9509 Write 2 Done Done New false 8511 Acked 9510 Read 7 Done Done New false 8512 Acked 9511 Read 4 Done Done New false 8513 Acked 9512 Write 6 Done Done New false 8514 Acked 9513 Read 6 Done Done New false 8515 Acked 9514 Write 10 Done Done New false 8516 Acked 9515 Read 9 Done Done New false 8517 Acked 9516 Read 9 Done Done New false 8518 Acked 9517 Read 3 Done Done New false 8519 Acked 9518 Write 9 Done Done New false 8520 Acked 9519 Read 8 Done Done New false 8521 Acked 9520 Write 5 Done Done New false 8522 Acked 9521 Read 2 Done Done New false 8523 Acked 9522 Write 4 Done Done New false 8524 Acked 9523 Flush 0 Done Done New false 8525 Acked 9524 Write 8 Done Done New false 8526 Acked 9525 Read 8 Done Done New false 8527 Acked 9526 Read 10 Done Done New false 8528 Acked 9527 Read 2 Done Done New false 8529 Acked 9528 Write 8 Done Done New false 8530 Acked 9529 Read 6 Done Done New false 8531 Acked 9530 Write 2 Done Done New false 8532 Acked 9531 Write 5 Done Done New false 8533 Acked 9532 Read 5 Done Done New false 8534 Acked 9533 Write 1 Done Done New false 8535 Acked 9534 Read 3 Done Done New false 8536 Acked 9535 Write 5 Done Done New false 8537 Acked 9536 Read 8 Done Done New false 8538 Acked 9537 Write 5 Done Done New false 8539 Acked 9538 Read 8 Done Done New false 8540 Acked 9539 Write 10 Done Done New false 8541 Acked 9540 Write 1 Done Done New false 8542 Acked 9541 Write 7 Done Done New false 8543 Acked 9542 Write 9 Done Done New false 8544 Acked 9543 Read 7 Done Done New false 8545 Acked 9544 Read 8 Done Done New false 8546 Acked 9545 Read 5 Done Done New false 8547 Acked 9546 Flush 0 Done Done New false 8548 Acked 9547 Write 5 Done Done New false 8549 Acked 9548 Read 8 Done Done New false 8550 Acked 9549 Read 8 Done Done New false 8551 Acked 9550 Flush 0 Done Done New false 8552 Acked 9551 Read 5 Done Done New false 8553 Acked 9552 Write 3 Done Done New false 8554 Acked 9553 Read 7 Done Done New false 8555 Acked 9554 Read 9 Done Done New false 8556 Acked 9555 Read 1 Done Done New false 8557 Acked 9556 Write 1 Done Done New false 8558 Acked 9557 Read 4 Done Done New false 8559 Acked 9558 Read 4 Done Done New false 8560 Acked 9559 Write 5 Done Done New false 8561 Acked 9560 Read 8 Done Done New false 8562 Acked 9561 Write 9 Done Done New false 8563 Acked 9562 Read 10 Done Done New false 8564 Acked 9563 Write 8 Done Done New false 8565 Acked 9564 Write 9 Done Done New false 8566 Acked 9565 Write 1 Done Done New false 8567 Acked 9566 Read 7 Done Done New false 8568 Acked 9567 Read 3 Done Done New false 8569 Acked 9568 Read 2 Done Done New false 8570 Acked 9569 Write 8 Done Done New false 8571 Acked 9570 Write 10 Done Done New false 8572 Acked 9571 Flush 0 Done Done New false 8573 Acked 9572 Write 3 Done Done New false 8574 Acked 9573 Write 2 Done Done New false 8575 Acked 9574 Read 1 Done Done New false 8576 Acked 9575 Read 7 Done Done New false 8577 Acked 9576 Read 9 Done Done New false 8578 Acked 9577 Read 8 Done Done New false 8579 Acked 9578 Read 5 Done Done New false 8580 Acked 9579 Write 4 Done Done New false 8581 Acked 9580 Write 9 Done Done New false 8582 Acked 9581 Write 10 Done Done New false 8583 Acked 9582 Write 7 Done Done New false 8584 Acked 9583 Flush 0 Done Done New false 8585 Acked 9584 Write 3 Done Done New false 8586 Acked 9585 Read 2 Done Done New false 8587 Acked 9586 Flush 0 Done Done New false 8588 Acked 9587 Read 9 Done Done New false 8589 Acked 9588 Write 2 Done Done New false 8590 Acked 9589 Write 9 Done Done New false 8591 Acked 9590 Read 8 Done Done New false 8592 Acked 9591 Read 5 Done Done New false 8593 Acked 9592 Read 10 Done Done New false 8594 Acked 9593 Write 9 Done Done New false 8595 Acked 9594 Read 7 Done Done New false 8596 Acked 9595 Write 6 Done Done New false 8597 Acked 9596 Read 7 Done Done New false 8598 Acked 9597 Read 8 Done Done New false 8599 Acked 9598 Read 6 Done Done New false 8600 Acked 9599 Read 5 Done Done New false 8601 Acked 9600 Write 7 Done Done New false 8602 Acked 9601 Read 10 Done Done New false 8603 Acked 9602 Write 1 Done Done New false 8604 Acked 9603 Write 9 Done Done New false 8605 Acked 9604 Read 2 Done Done New false 8606 Acked 9605 Read 4 Done Done New false 8607 Acked 9606 Read 3 Done Done New false 8608 Acked 9607 Write 10 Done Done New false 8609 Acked 9608 Write 5 Done Done New false 8610 Acked 9609 Write 7 Done Done New false 8611 Acked 9610 Flush 0 Done Done New false 8612 Acked 9611 Write 8 Done Done New false 8613 Acked 9612 Read 1 Done Done New false 8614 Acked 9613 Write 8 Done Done New false 8615 Acked 9614 Write 1 Done Done New false 8616 Acked 9615 Read 9 Done Done New false 8617 Acked 9616 Read 8 Done Done New false 8618 Acked 9617 Read 9 Done Done New false 8619 Acked 9618 Write 1 Done Done New false 8620 Acked 9619 Write 8 Done Done New false 8621 Acked 9620 Write 10 Done Done New false 8622 Acked 9621 Write 8 Done Done New false 8623 Acked 9622 Write 8 Done Done New false 8624 Acked 9623 Write 3 Done Done New false 8625 Acked 9624 Read 2 Done Done New false 8626 Acked 9625 Read 7 Done Done New false 8627 Acked 9626 Read 5 Done Done New false 8628 Acked 9627 Write 8 Done Done New false 8629 Acked 9628 Write 10 Done Done New false 8630 Acked 9629 Read 7 Done Done New false 8631 Acked 9630 Read 4 Done Done New false 8632 Acked 9631 Read 6 Done Done New false 8633 Acked 9632 Read 6 Done Done New false 8634 Acked 9633 Write 4 Done Done New false 8635 Acked 9634 Flush 0 Done Done New false 8636 Acked 9635 Write 1 Done Done New false 8637 Acked 9636 Write 9 Done Done New false 8638 Acked 9637 Write 9 Done Done New false 8639 Acked 9638 Read 3 Done Done New false 8640 Acked 9639 Write 2 Done Done New false 8641 Acked 9640 Write 6 Done Done New false 8642 Acked 9641 Read 3 Done Done New false 8643 Acked 9642 Read 5 Done Done New false 8644 Acked 9643 Write 3 Done Done New false 8645 Acked 9644 Write 6 Done Done New false 8646 Acked 9645 Write 6 Done Done New false 8647 Acked 9646 Flush 0 Done Done New false 8648 Acked 9647 Read 6 Done Done New false 8649 Acked 9648 Write 3 Done Done New false 8650 Acked 9649 Read 6 Done Done New false 8651 Acked 9650 Write 1 Done Done New false 8652 Acked 9651 Flush 0 Done Done New false 8653 Acked 9652 Read 5 Done Done New false 8654 Acked 9653 Read 3 Done Done New false 8655 Acked 9654 Write 8 Done Done New false 8656 Acked 9655 Write 9 Done Done New false 8657 Acked 9656 Read 9 Done Done New false 8658 Acked 9657 Read 3 Done Done New false 8659 Acked 9658 Read 10 Done Done New false 8660 Acked 9659 Flush 0 Done Done New false 8661 Acked 9660 Read 3 Done Done New false 8662 Acked 9661 Write 10 Done Done New false 8663 Acked 9662 Write 9 Done Done New false 8664 Acked 9663 Flush 0 Done Done New false 8665 Acked 9664 Read 1 Done Done New false 8666 Acked 9665 Write 7 Done Done New false 8667 Acked 9666 Read 2 Done Done New false 8668 Acked 9667 Write 3 Done Done New false 8669 Acked 9668 Read 9 Done Done New false 8670 Acked 9669 Read 2 Done Done New false 8671 Acked 9670 Read 4 Done Done New false 8672 Acked 9671 Write 1 Done Done New false 8673 Acked 9672 Read 3 Done Done New false 8674 Acked 9673 Write 9 Done Done New false 8675 Acked 9674 Write 1 Done Done New false 8676 Acked 9675 Write 2 Done Done New false 8677 Acked 9676 Read 5 Done Done New false 8678 Acked 9677 Read 8 Done Done New false 8679 Acked 9678 Flush 0 Done Done New false 8680 Acked 9679 Read 8 Done Done New false 8681 Acked 9680 Flush 0 Done Done New false 8682 Acked 9681 Write 4 Done Done New false 8683 Acked 9682 Read 2 Done Done New false 8684 Acked 9683 Read 1 Done Done New false 8685 Acked 9684 Write 2 Done Done New false 8686 Acked 9685 Read 10 Done Done New false 8687 Acked 9686 Write 1 Done Done New false 8688 Acked 9687 Read 4 Done Done New false 8689 Acked 9688 Read 8 Done Done New false 8690 Acked 9689 Write 3 Done Done New false 8691 Acked 9690 Write 8 Done Done New false 8692 Acked 9691 Read 5 Done Done New false 8693 Acked 9692 Write 7 Done Done New false 8694 Acked 9693 Write 9 Done Done New false 8695 Acked 9694 Write 4 Done Done New false 8696 Acked 9695 Write 5 Done Done New false 8697 Acked 9696 Read 1 Done Done New false 8698 Acked 9697 Read 6 Done Done New false 8699 Acked 9698 Read 2 Done Done New false 8700 Acked 9699 Write 2 Done Done New false 8701 Acked 9700 Read 7 Done Done New false 8702 Acked 9701 Read 5 Done Done New false 8703 Acked 9702 Read 8 Done Done New false 8704 Acked 9703 Flush 0 Done Done New false 8705 Acked 9704 Read 5 Done Done New false 8706 Acked 9705 Read 7 Done Done New false 8707 Acked 9706 Write 9 Done Done New false 8708 Acked 9707 Write 7 Done Done New false 8709 Acked 9708 Read 9 Done Done New false 8710 Acked 9709 Write 4 Done Done New false 8711 Acked 9710 Read 9 Done Done New false 8712 Acked 9711 Write 1 Done Done New false 8713 Acked 9712 Write 3 Done Done New false 8714 Acked 9713 Write 7 Done Done New false 8715 Acked 9714 Read 6 Done Done New false 8716 Acked 9715 Read 7 Done Done New false 8717 Acked 9716 Read 4 Done Done New false 8718 Acked 9717 Read 7 Done Done New false 8719 Acked 9718 Write 10 Done Done New false 8720 Acked 9719 Flush 0 Done Done New false 8721 Acked 9720 Read 4 Done Done New false 8722 Acked 9721 Write 10 Done Done New false 8723 Acked 9722 Write 3 Done Done New false 8724 Acked 9723 Read 3 Done Done New false 8725 Acked 9724 Read 1 Done Done New false 8726 Acked 9725 Read 10 Done Done New false 8727 Acked 9726 Write 1 Done Done New false 8728 Acked 9727 Write 2 Done Done New false 8729 Acked 9728 Read 8 Done Done New false 8730 Acked 9729 Read 5 Done Done New false 8731 Acked 9730 Write 3 Done Done New false 8732 Acked 9731 Read 7 Done Done New false 8733 Acked 9732 Write 8 Done Done New false 8734 Acked 9733 Write 10 Done Done New false 8735 Acked 9734 Read 9 Done Done New false 8736 Acked 9735 Read 5 Done Done New false 8737 Acked 9736 Write 7 Done Done New false 8738 Acked 9737 Read 3 Done Done New false 8739 Acked 9738 Read 5 Done Done New false 8740 Acked 9739 Flush 0 Done Done New false 8741 Acked 9740 Read 9 Done Done New false 8742 Acked 9741 Flush 0 Done Done New false 8743 Acked 9742 Write 9 Done Done New false 8744 Acked 9743 Read 4 Done Done New false 8745 Acked 9744 Read 7 Done Done New false 8746 Acked 9745 Write 1 Done Done New false 8747 Acked 9746 Write 2 Done Done New false 8748 Acked 9747 Write 9 Done Done New false 8749 Acked 9748 Write 1 Done Done New false 8750 Acked 9749 Read 4 Done Done New false 8751 Acked 9750 Write 5 Done Done New false 8752 Acked 9751 Flush 0 Done Done New false 8753 Acked 9752 Read 10 Done Done New false 8754 Acked 9753 Read 4 Done Done New false 8755 Acked 9754 Write 2 Done Done New false 8756 Acked 9755 Write 1 Done Done New false 8757 Acked 9756 Read 10 Done Done New false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 301 301 Sent 0 0 0 0 Done 301 301 0 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9751 9751 9455 Downstairs last five completed: 9455 9454 9453 9452 9451 Upstairs last five completed: 8757 8756 8755 8754 8753 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:16.038677903Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:16.038891393Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:16.038946814Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:16.039039344Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:16.039060483Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:16.039082883Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:16.039189806Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:17.041106939Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:17.041177304Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:17.04120528Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:17.041229835Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:17.041256306Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:17.041286592Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] client re-new 302 jobs since flush 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:17.041310207Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:17.041536669Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8830] got start action [8830] I am going to restart Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] reports Starting Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1214) [8830][2] reports Running {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.042530003Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.042717754Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.044716618Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.044764368Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.04479162Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 9455","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.045066518Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.045101096Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.045124681Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.045147063Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:18.045169376Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"2","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 9757 9756 9755 9754 9753 Upstairs last five completed: 8758 8757 8756 8755 8754 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Main task has work to do, go find it got dsc Stop(0) stop 0 [8810] Got stop action so:false kr:false Jan 02 22:36:19.652 INFO request completed, latency_us: 465, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 32c05faf-0266-465a-b066-c46549936dbc, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Jan 02 22:36:19.653 INFO request completed, latency_us: 222, response_code: 200, uri: /state/cid/0, method: GET, req_id: 396a2535-7685-40ec-ae9d-6f4069b85fed, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:19.654020894Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:19.654210618Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:19.654258483Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:19.654281944Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:19.654394125Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:19.654500429Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:20.655816784Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:20.655932481Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:20.655993864Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:20.656047642Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:20.656067974Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:20.656090357Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:20.656123356Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:21.658044425Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:21.658220573Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:21.658351112Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"exiting client task is ignoring message Ruok","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:21.658378013Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:21.658421968Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:21.65853466Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:21.658568538Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:21.658686032Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:22.659160094Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:22.659227474Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:22.659254941Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:22.659281319Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:22.65931287Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 0 jobs since flush 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:22.659336645Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:22.659367135Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Jan 02 22:36:23.656 INFO request completed, latency_us: 247, response_code: 200, uri: /state/cid/0, method: GET, req_id: b154edeb-1194-4553-89c1-8e0607ec48b3, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 001/300 Write block 4376 len 16384 data: 5 5 5 7 002/300 Write block 4287 len 28672 data: 5 6 5 6 6 6 6 003/300 Flush {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:23.660930017Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:23.661074902Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:23.661105902Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:23.661132758Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:23.661199617Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 3 jobs since flush 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:23.661247006Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:23.661285098Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} 004/300 Write block 798 len 36864 data: 4 5 5 7 7 8 5 6 6 005/300 Read block 4234 len 4096 006/300 Read block 1093 len 8192 007/300 Read block 903 len 36864 008/300 Flush 009/300 Read block 362 len 40960 010/300 Write block 3487 len 36864 data: 5 4 5 5 4 4 4 6 5 011/300 Read block 1837 len 36864 012/300 Write block 2850 len 8192 data: 6 7 013/300 Read block 2601 len 4096 014/300 Read block 4133 len 20480 015/300 Read block 2066 len 8192 016/300 Write block 4693 len 4096 data: 5 017/300 Write block 2588 len 4096 data: 6 018/300 Write block 1347 len 4096 data: 8 019/300 Flush 020/300 Read block 1363 len 4096 021/300 Read block 3205 len 28672 022/300 Write block 1835 len 36864 data: 4 4 5 6 6 6 6 5 6 023/300 Read block 4201 len 32768 024/300 Read block 3483 len 4096 025/300 Write block 2893 len 36864 data: 5 6 6 8 8 9 7 6 6 026/300 Write block 2361 len 24576 data: 4 5 7 6 7 7 027/300 Write block 153 len 36864 data: 6 5 6 4 4 4 5 6 5 028/300 Read block 4729 len 32768 029/300 Write block 764 len 36864 data: 3 3 3 3 2 2 3 4 5 030/300 Read block 4226 len 40960 031/300 Read block 282 len 8192 032/300 Read block 4412 len 12288 033/300 Read block 1974 len 36864 034/300 Read block 1198 len 16384 035/300 Write block 1161 len 16384 data: 6 8 8 7 036/300 Read block 1983 len 32768 037/300 Read block 632 len 36864 038/300 Read block 3744 len 12288 039/300 Write block 216 len 40960 data: 8 7 6 5 5 6 7 7 8 8 040/300 Read block 1651 len 24576 041/300 Flush 042/300 Write block 1403 len 28672 data: 5 4 5 3 3 3 3 043/300 Write block 4667 len 8192 data: 5 4 044/300 Flush 045/300 Read block 1709 len 4096 046/300 Write block 2894 len 16384 data: 7 7 9 9 047/300 Read block 4344 len 4096 048/300 Read block 3205 len 16384 049/300 Read block 380 len 4096 050/300 Write block 1179 len 8192 data: 6 6 051/300 Flush 052/300 Read block 1258 len 16384 053/300 Read block 4404 len 32768 054/300 Read block 3515 len 8192 055/300 Write block 4052 len 16384 data: 6 5 5 5 056/300 Read block 2548 len 16384 057/300 Read block 3958 len 8192 058/300 Read block 664 len 36864 059/300 Write block 2706 len 28672 data: 6 4 5 5 6 5 4 060/300 Write block 3961 len 16384 data: 8 9 8 8 061/300 Write block 3291 len 4096 data: 3 062/300 Read block 526 len 36864 063/300 Read block 638 len 4096 064/300 Read block 4328 len 36864 065/300 Flush 066/300 Write block 289 len 36864 data: 9 7 6 7 7 8 8 8 9 067/300 Read block 1399 len 12288 068/300 Write block 3328 len 8192 data: 4 3 069/300 Write block 906 len 40960 data: 4 5 4 4 3 5 4 4 4 3 070/300 Read block 1601 len 12288 071/300 Write block 2600 len 32768 data: 6 5 4 5 5 6 5 5 072/300 Write block 1701 len 4096 data: 4 073/300 Write block 156 len 28672 data: 5 5 5 6 7 6 5 074/300 Write block 1171 len 8192 data: 4 3 075/300 Write block 2562 len 16384 data: 5 6 6 6 076/300 Read block 3581 len 8192 077/300 Write block 515 len 24576 data: 9 11 8 9 7 6 078/300 Read block 2493 len 28672 079/300 Flush 080/300 Read block 3998 len 32768 081/300 Write block 1228 len 24576 data: 7 8 7 7 7 6 082/300 Read block 2243 len 12288 083/300 Read block 4732 len 32768 084/300 Read block 2878 len 16384 085/300 Write block 404 len 20480 data: 8 6 4 5 4 086/300 Read block 2875 len 28672 087/300 Write block 2895 len 4096 data: 8 088/300 Flush 089/300 Read block 2125 len 40960 090/300 Read block 3276 len 4096 091/300 Write block 2421 len 24576 data: 7 6 7 6 4 3 092/300 Flush 093/300 Write block 4744 len 40960 data: 6 6 6 7 7 8 7 6 5 6 094/300 Write block 4367 len 32768 data: 5 6 6 5 5 5 5 5 095/300 Read block 4136 len 40960 096/300 Write block 4659 len 36864 data: 3 3 4 4 5 5 6 6 6 097/300 Read block 1074 len 4096 098/300 Read block 2394 len 8192 099/300 Write block 4414 len 40960 data: 5 5 5 6 8 9 8 7 8 7 100/300 Read block 4992 len 12288 101/300 Read block 2608 len 40960 102/300 Flush 103/300 Read block 3606 len 32768 104/300 Write block 4943 len 12288 data: 5 5 5 105/300 Read block 4288 len 8192 106/300 Read block 3744 len 36864 107/300 Write block 3450 len 12288 data: 3 5 6 108/300 Read block 1190 len 12288 109/300 Write block 1907 len 32768 data: 5 4 4 5 7 8 7 5 110/300 Read block 1694 len 40960 111/300 Write block 1772 len 28672 data: 8 7 6 8 7 5 4 112/300 Write block 2703 len 16384 data: 4 4 6 7 113/300 Write block 3464 len 36864 data: 2 5 5 5 6 5 7 5 5 114/300 Read block 3045 len 4096 115/300 Write block 796 len 40960 data: 6 6 5 6 6 8 8 9 6 7 116/300 Read block 4912 len 12288 117/300 Read block 2540 len 24576 118/300 Read block 4508 len 32768 119/300 Flush 120/300 Write block 3543 len 20480 data: 10 9 6 5 5 121/300 Read block 4046 len 40960 122/300 Read block 3397 len 40960 123/300 Read block 3090 len 40960 124/300 Write block 4681 len 40960 data: 4 5 5 5 5 6 4 4 4 4 125/300 Read block 865 len 32768 126/300 Write block 220 len 28672 data: 6 7 8 8 9 9 7 127/300 Read block 738 len 24576 128/300 Write block 4925 len 40960 data: 12 13 13 11 11 10 13 10 10 9 129/300 Write block 295 len 4096 data: 9 130/300 Write block 568 len 4096 data: 2 131/300 Write block 2617 len 8192 data: 7 7 132/300 Read block 4649 len 36864 133/300 Read block 4957 len 16384 134/300 Read block 1220 len 28672 135/300 Write block 2314 len 24576 data: 7 5 5 5 4 4 136/300 Write block 4866 len 24576 data: 5 5 5 8 6 6 137/300 Read block 1282 len 40960 138/300 Read block 4948 len 32768 139/300 Read block 3052 len 40960 140/300 Read block 1571 len 16384 141/300 Flush 142/300 Read block 2708 len 12288 143/300 Read block 3054 len 12288 144/300 Write block 3128 len 20480 data: 4 6 5 5 7 145/300 Read block 3244 len 16384 146/300 Write block 2608 len 4096 data: 6 147/300 Read block 1322 len 4096 148/300 Flush 149/300 Flush 150/300 Read block 1196 len 12288 151/300 Read block 864 len 28672 152/300 Read block 4942 len 4096 153/300 Read block 3029 len 16384 154/300 Write block 3457 len 40960 data: 9 7 6 5 6 4 3 3 6 6 155/300 Write block 3547 len 20480 data: 6 4 5 5 5 156/300 Read block 291 len 4096 157/300 Write block 1461 len 36864 data: 9 7 8 5 7 6 7 7 7 158/300 Read block 3088 len 12288 159/300 Read block 2573 len 12288 160/300 Read block 2644 len 4096 161/300 Write block 2935 len 16384 data: 7 8 4 3 162/300 Write block 973 len 16384 data: 8 6 7 8 163/300 Read block 1026 len 40960 164/300 Read block 243 len 20480 165/300 Read block 4733 len 4096 166/300 Read block 4553 len 4096 167/300 Write block 2828 len 40960 data: 9 8 7 5 4 4 5 7 7 7 168/300 Flush 169/300 Read block 2783 len 12288 170/300 Write block 3444 len 24576 data: 3 3 3 4 4 4 171/300 Read block 2724 len 12288 172/300 Read block 568 len 28672 173/300 Read block 2777 len 32768 174/300 Read block 3802 len 4096 175/300 Read block 2592 len 20480 176/300 Write block 1156 len 12288 data: 4 4 3 177/300 Write block 2769 len 12288 data: 10 11 10 178/300 Write block 2180 len 20480 data: 5 4 5 5 3 179/300 Write block 4754 len 8192 data: 6 6 180/300 Read block 4218 len 32768 181/300 Write block 2914 len 40960 data: 6 5 6 8 6 6 7 6 5 5 182/300 Write block 3358 len 12288 data: 8 10 10 183/300 Read block 1264 len 12288 184/300 Read block 4431 len 16384 185/300 Write block 2813 len 40960 data: 3 3 4 7 6 8 8 9 12 13 186/300 Write block 1168 len 20480 data: 6 7 6 5 4 187/300 Flush 188/300 Write block 4093 len 20480 data: 10 11 7 7 7 189/300 Read block 4539 len 12288 190/300 Write block 2307 len 28672 data: 7 8 8 7 6 7 6 191/300 Read block 174 len 36864 192/300 Write block 3700 len 36864 data: 6 6 6 4 3 4 4 5 5 193/300 Flush 194/300 Flush 195/300 Flush 196/300 Write block 1965 len 16384 data: 5 5 5 5 197/300 Write block 2306 len 12288 data: 8 8 9 198/300 Write block 3763 len 40960 data: 9 9 11 12 11 11 9 8 8 5 199/300 Read block 2512 len 24576 200/300 Write block 1796 len 16384 data: 5 4 3 4 201/300 Read block 3401 len 32768 202/300 Read block 4304 len 12288 203/300 Read block 1627 len 4096 204/300 Flush 205/300 Read block 1233 len 12288 206/300 Write block 3484 len 36864 data: 7 6 5 6 5 6 6 5 5 207/300 Write block 696 len 20480 data: 7 5 5 5 4 208/300 Write block 468 len 20480 data: 4 6 7 6 7 209/300 Read block 1397 len 12288 210/300 Read block 899 len 4096 211/300 Read block 2554 len 24576 212/300 Read block 3397 len 28672 213/300 Write block 2823 len 32768 data: 14 11 11 11 10 10 9 8 214/300 Read block 3902 len 24576 215/300 Read block 3413 len 36864 216/300 Read block 344 len 40960 217/300 Flush 218/300 Read block 3559 len 4096 219/300 Write block 228 len 36864 data: 6 5 4 4 4 8 8 7 7 220/300 Write block 668 len 40960 data: 5 5 7 7 6 5 6 7 7 6 221/300 Read block 504 len 16384 222/300 Write block 3422 len 4096 data: 6 223/300 Read block 370 len 36864 224/300 Read block 10 len 4096 225/300 Flush 226/300 Read block 3985 len 32768 227/300 Read block 1469 len 20480 228/300 Write block 3988 len 40960 data: 9 9 8 9 7 10 9 10 12 10 229/300 Flush 230/300 Write block 46 len 24576 data: 6 7 8 8 6 5 231/300 Read block 2068 len 40960 232/300 Read block 2443 len 32768 233/300 Read block 2781 len 32768 234/300 Write block 4363 len 16384 data: 7 6 6 4 235/300 Read block 2486 len 12288 236/300 Write block 4154 len 24576 data: 6 6 5 6 4 4 237/300 Write block 3496 len 4096 data: 6 238/300 Read block 663 len 4096 239/300 Read block 2245 len 40960 240/300 Read block 2889 len 24576 241/300 Read block 4221 len 36864 242/300 Write block 1753 len 8192 data: 6 7 243/300 Flush 244/300 Read block 2864 len 20480 245/300 Flush 246/300 Read block 2933 len 20480 247/300 Write block 1320 len 16384 data: 4 6 8 9 248/300 Read block 3335 len 28672 249/300 Read block 1796 len 4096 250/300 Read block 2297 len 16384 251/300 Flush 252/300 Flush 253/300 Write block 2039 len 24576 data: 4 4 3 4 5 5 254/300 Read block 1281 len 20480 255/300 Write block 3732 len 16384 data: 5 5 5 5 256/300 Write block 282 len 4096 data: 6 257/300 Read block 4834 len 28672 258/300 Read block 978 len 12288 259/300 Read block 1470 len 24576 260/300 Write block 2891 len 20480 data: 3 4 6 8 9 261/300 Write block 2215 len 16384 data: 5 3 4 4 262/300 Read block 4441 len 8192 263/300 Read block 3678 len 16384 264/300 Write block 4941 len 12288 data: 4 5 6 265/300 Read block 2493 len 32768 266/300 Write block 4111 len 24576 data: 3 3 4 4 4 4 267/300 Write block 2989 len 36864 data: 6 6 6 5 5 2 3 3 3 268/300 Read block 197 len 28672 269/300 Read block 632 len 12288 270/300 Read block 1988 len 36864 271/300 Write block 1767 len 16384 data: 6 6 6 5 272/300 Read block 1104 len 12288 273/300 Write block 841 len 28672 data: 4 5 6 5 6 7 7 274/300 Flush 275/300 Flush 276/300 Read block 3539 len 4096 277/300 Read block 1763 len 24576 278/300 Flush 279/300 Read block 3721 len 16384 280/300 Read block 3390 len 8192 281/300 Write block 147 len 28672 data: 8 10 8 7 6 6 7 282/300 Read block 1969 len 4096 283/300 Read block 2905 len 40960 284/300 Read block 3222 len 28672 285/300 Read block 2140 len 20480 286/300 Read block 2261 len 36864 287/300 Write block 60 len 4096 data: 5 288/300 Write block 1436 len 20480 data: 4 4 4 3 4 289/300 Write block 1449 len 8192 data: 5 5 290/300 Read block 3492 len 4096 291/300 Write block 597 len 12288 data: 5 6 7 292/300 Flush 293/300 Read block 381 len 12288 294/300 Write block 983 len 36864 data: 4 3 4 3 4 3 4 4 4 295/300 Read block 3007 len 4096 296/300 Write block 4502 len 12288 data: 6 8 8 297/300 Read block 2535 len 32768 298/300 Read block 2037 len 28672 299/300 Read block 333 len 12288 300/300 Read block 4416 len 28672 Main task has work to do, go find it got dsc Start(0) start 0 Jan 02 22:36:24.298 INFO request completed, latency_us: 293, response_code: 204, uri: /start/cid/0, method: POST, req_id: 331d1ce2-239f-4b43-af82-cf43bbbe2188, remote_addr: 127.0.0.1:37434, local_addr: 127.0.0.1:9998 Replay: started 0, returned:Ok(()) ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301 States: Offline Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 8759 Acked 9758 Write 4 New Done Done true 8760 Acked 9759 Write 7 New Done Done true 8761 Acked 9760 Flush 0 New Done Done true 8762 Acked 9761 Write 9 New Done Done false 8763 Acked 9762 Read 1 New Done Done false 8764 Acked 9763 Read 2 New Done Done false 8765 Acked 9764 Read 9 New Done Done false 8766 Acked 9765 Flush 0 New Done Done false 8767 Acked 9766 Read 10 New Done Done false 8768 Acked 9767 Write 9 New Done Done false 8769 Acked 9768 Read 9 New Done Done false 8770 Acked 9769 Write 2 New Done Done false 8771 Acked 9770 Read 1 New Done Done false 8772 Acked 9771 Read 5 New Done Done false 8773 Acked 9772 Read 2 New Done Done false 8774 Acked 9773 Write 1 New Done Done false 8775 Acked 9774 Write 1 New Done Done false 8776 Acked 9775 Write 1 New Done Done false 8777 Acked 9776 Flush 0 New Done Done false 8778 Acked 9777 Read 1 New Done Done false 8779 Acked 9778 Read 7 New Done Done false 8780 Acked 9779 Write 9 New Done Done false 8781 Acked 9780 Read 8 New Done Done false 8782 Acked 9781 Read 1 New Done Done false 8783 Acked 9782 Write 9 New Done Done false 8784 Acked 9783 Write 6 New Done Done false 8785 Acked 9784 Write 9 New Done Done false 8786 Acked 9785 Read 8 New Done Done false 8787 Acked 9786 Write 9 New Done Done false 8788 Acked 9787 Read 10 New Done Done false 8789 Acked 9788 Read 2 New Done Done false 8790 Acked 9789 Read 3 New Done Done false 8791 Acked 9790 Read 9 New Done Done false 8792 Acked 9791 Read 4 New Done Done false 8793 Acked 9792 Write 4 New Done Done false 8794 Acked 9793 Read 8 New Done Done false 8795 Acked 9794 Read 9 New Done Done false 8796 Acked 9795 Read 3 New Done Done false 8797 Acked 9796 Write 10 New Done Done false 8798 Acked 9797 Read 6 New Done Done false 8799 Acked 9798 Flush 0 New Done Done false 8800 Acked 9799 Write 7 New Done Done false 8801 Acked 9800 Write 2 New Done Done false 8802 Acked 9801 Flush 0 New Done Done false 8803 Acked 9802 Read 1 New Done Done false 8804 Acked 9803 Write 4 New Done Done false 8805 Acked 9804 Read 1 New Done Done false 8806 Acked 9805 Read 4 New Done Done false 8807 Acked 9806 Read 1 New Done Done false 8808 Acked 9807 Write 2 New Done Done false 8809 Acked 9808 Flush 0 New Done Done false 8810 Acked 9809 Read 4 New Done Done false 8811 Acked 9810 Read 8 New Done Done false 8812 Acked 9811 Read 2 New Done Done false 8813 Acked 9812 Write 4 New Done Done false 8814 Acked 9813 Read 4 New Done Done false 8815 Acked 9814 Read 2 New Done Done false 8816 Acked 9815 Read 9 New Done Done false 8817 Acked 9816 Write 7 New Done Done false 8818 Acked 9817 Write 4 New Done Done false 8819 Acked 9818 Write 1 New Done Done false 8820 Acked 9819 Read 9 New Done Done false 8821 Acked 9820 Read 1 New Done Done false 8822 Acked 9821 Read 9 New Done Done false 8823 Acked 9822 Flush 0 New Done Done false 8824 Acked 9823 Write 9 New Done Done false 8825 Acked 9824 Read 3 New Done Done false 8826 Acked 9825 Write 2 New Done Done false 8827 Acked 9826 Write 10 New Done Done false 8828 Acked 9827 Read 3 New Done Done false 8829 Acked 9828 Write 8 New Done Done false 8830 Acked 9829 Write 1 New Done Done false 8831 Acked 9830 Write 7 New Done Done false 8832 Acked 9831 Write 2 New Done Done false 8833 Acked 9832 Write 4 New Done Done false 8834 Acked 9833 Read 2 New Done Done false 8835 Acked 9834 Write 6 New Done Done false 8836 Acked 9835 Read 7 New Done Done false 8837 Acked 9836 Flush 0 New Done Done false 8838 Acked 9837 Read 8 New Done Done false 8839 Acked 9838 Write 6 New Done Done false 8840 Acked 9839 Read 3 New Done Done false 8841 Acked 9840 Read 8 New Done Done false 8842 Acked 9841 Read 4 New Done Done false 8843 Acked 9842 Write 5 New Done Done false 8844 Acked 9843 Read 7 New Done Done false 8845 Acked 9844 Write 1 New Done Done false 8846 Acked 9845 Flush 0 New Done Done false 8847 Acked 9846 Read 10 New Done Done false 8848 Acked 9847 Read 1 New Done Done false 8849 Acked 9848 Write 6 New Done Done false 8850 Acked 9849 Flush 0 New Done Done false 8851 Acked 9850 Write 10 New Done Done false 8852 Acked 9851 Write 8 New Done Done false 8853 Acked 9852 Read 10 New Done Done false 8854 Acked 9853 Write 9 New Done Done false 8855 Acked 9854 Read 1 New Done Done false 8856 Acked 9855 Read 2 New Done Done false 8857 Acked 9856 Write 10 New Done Done false 8858 Acked 9857 Read 3 New Done Done false 8859 Acked 9858 Read 10 New Done Done false 8860 Acked 9859 Flush 0 New Done Done false 8861 Acked 9860 Read 8 New Done Done false 8862 Acked 9861 Write 3 New Done Done false 8863 Acked 9862 Read 2 New Done Done false 8864 Acked 9863 Read 9 New Done Done false 8865 Acked 9864 Write 3 New Done Done false 8866 Acked 9865 Read 3 New Done Done false 8867 Acked 9866 Write 8 New Done Done false 8868 Acked 9867 Read 10 New Done Done false 8869 Acked 9868 Write 7 New Done Done false 8870 Acked 9869 Write 4 New Done Done false 8871 Acked 9870 Write 9 New Done Done false 8872 Acked 9871 Read 1 New Done Done false 8873 Acked 9872 Write 10 New Done Done false 8874 Acked 9873 Read 3 New Done Done false 8875 Acked 9874 Read 6 New Done Done false 8876 Acked 9875 Read 8 New Done Done false 8877 Acked 9876 Flush 0 New Done Done false 8878 Acked 9877 Write 5 New Done Done false 8879 Acked 9878 Read 10 New Done Done false 8880 Acked 9879 Read 10 New Done Done false 8881 Acked 9880 Read 10 New Done Done false 8882 Acked 9881 Write 10 New Done Done false 8883 Acked 9882 Read 8 New Done Done false 8884 Acked 9883 Write 7 New Done Done false 8885 Acked 9884 Read 6 New Done Done false 8886 Acked 9885 Write 10 New Done Done false 8887 Acked 9886 Write 1 New Done Done false 8888 Acked 9887 Write 1 New Done Done false 8889 Acked 9888 Write 2 New Done Done false 8890 Acked 9889 Read 9 New Done Done false 8891 Acked 9890 Read 4 New Done Done false 8892 Acked 9891 Read 7 New Done Done false 8893 Acked 9892 Write 6 New Done Done false 8894 Acked 9893 Write 6 New Done Done false 8895 Acked 9894 Read 10 New Done Done false 8896 Acked 9895 Read 8 New Done Done false 8897 Acked 9896 Read 10 New Done Done false 8898 Acked 9897 Read 4 New Done Done false 8899 Acked 9898 Flush 0 New Done Done false 8900 Acked 9899 Read 3 New Done Done false 8901 Acked 9900 Read 3 New Done Done false 8902 Acked 9901 Write 5 New Done Done false 8903 Acked 9902 Read 4 New Done Done false 8904 Acked 9903 Write 1 New Done Done false 8905 Acked 9904 Read 1 New Done Done false 8906 Acked 9905 Flush 0 New Done Done false 8907 Acked 9906 Flush 0 New Done Done false 8908 Acked 9907 Read 3 New Done Done false 8909 Acked 9908 Read 7 New Done Done false 8910 Acked 9909 Read 1 New Done Done false 8911 Acked 9910 Read 4 New Done Done false 8912 Acked 9911 Write 10 New Done Done false 8913 Acked 9912 Write 5 New Done Done false 8914 Acked 9913 Read 1 New Done Done false 8915 Acked 9914 Write 9 New Done Done false 8916 Acked 9915 Read 3 New Done Done false 8917 Acked 9916 Read 3 New Done Done false 8918 Acked 9917 Read 1 New Done Done false 8919 Acked 9918 Write 4 New Done Done false 8920 Acked 9919 Write 4 New Done Done false 8921 Acked 9920 Read 10 New Done Done false 8922 Acked 9921 Read 5 New Done Done false 8923 Acked 9922 Read 1 New Done Done false 8924 Acked 9923 Read 1 New Done Done false 8925 Acked 9924 Write 10 New Done Done false 8926 Acked 9925 Flush 0 New Done Done false 8927 Acked 9926 Read 3 New Done Done false 8928 Acked 9927 Write 6 New Done Done false 8929 Acked 9928 Read 3 New Done Done false 8930 Acked 9929 Read 7 New Done Done false 8931 Acked 9930 Read 8 New Done Done false 8932 Acked 9931 Flush 0 New Done Done false 8933 Acked 9932 Read 1 New Done Done false 8934 Acked 9933 Read 5 New Done Done false 8935 Acked 9934 Write 3 New Done Done false 8936 Acked 9935 Write 3 New Done Done false 8937 Acked 9936 Write 5 New Done Done false 8938 Acked 9937 Write 2 New Done Done false 8939 Acked 9938 Read 8 New Done Done false 8940 Acked 9939 Write 10 New Done Done false 8941 Acked 9940 Write 3 New Done Done false 8942 Acked 9941 Read 3 New Done Done false 8943 Acked 9942 Read 4 New Done Done false 8944 Acked 9943 Write 10 New Done Done false 8945 Acked 9944 Write 5 New Done Done false 8946 Acked 9945 Flush 0 New Done Done false 8947 Acked 9946 Write 5 New Done Done false 8948 Acked 9947 Read 3 New Done Done false 8949 Acked 9948 Write 7 New Done Done false 8950 Acked 9949 Read 9 New Done Done false 8951 Acked 9950 Write 9 New Done Done false 8952 Acked 9951 Flush 0 New Done Done false 8953 Acked 9952 Flush 0 New Done Done false 8954 Acked 9953 Flush 0 New Done Done false 8955 Acked 9954 Write 4 New Done Done false 8956 Acked 9955 Write 3 New Done Done false 8957 Acked 9956 Write 10 New Done Done false 8958 Acked 9957 Read 6 New Done Done false 8959 Acked 9958 Write 4 New Done Done false 8960 Acked 9959 Read 8 New Done Done false 8961 Acked 9960 Read 3 New Done Done false 8962 Acked 9961 Read 1 New Done Done false 8963 Acked 9962 Flush 0 New Done Done false 8964 Acked 9963 Read 3 New Done Done false 8965 Acked 9964 Write 9 New Done Done false 8966 Acked 9965 Write 5 New Done Done false 8967 Acked 9966 Write 5 New Done Done false 8968 Acked 9967 Read 3 New Done Done false 8969 Acked 9968 Read 1 New Done Done false 8970 Acked 9969 Read 6 New Done Done false 8971 Acked 9970 Read 7 New Done Done false 8972 Acked 9971 Write 8 New Done Done false 8973 Acked 9972 Read 6 New Done Done false 8974 Acked 9973 Read 9 New Done Done false 8975 Acked 9974 Read 10 New Done Done false 8976 Acked 9975 Flush 0 New Done Done false 8977 Acked 9976 Read 1 New Done Done false 8978 Acked 9977 Write 9 New Done Done false 8979 Acked 9978 Write 10 New Done Done false 8980 Acked 9979 Read 4 New Done Done false 8981 Acked 9980 Write 1 New Done Done false 8982 Acked 9981 Read 9 New Done Done false 8983 Acked 9982 Read 1 New Done Done false 8984 Acked 9983 Flush 0 New Done Done false 8985 Acked 9984 Read 8 New Done Done false 8986 Acked 9985 Read 5 New Done Done false 8987 Acked 9986 Write 10 New Done Done false 8988 Acked 9987 Flush 0 New Done Done false 8989 Acked 9988 Write 6 New Done Done false 8990 Acked 9989 Read 10 New Done Done false 8991 Acked 9990 Read 8 New Done Done false 8992 Acked 9991 Read 8 New Done Done false 8993 Acked 9992 Write 4 New Done Done false 8994 Acked 9993 Read 3 New Done Done false 8995 Acked 9994 Write 6 New Done Done false 8996 Acked 9995 Write 1 New Done Done false 8997 Acked 9996 Read 1 New Done Done false 8998 Acked 9997 Read 10 New Done Done false 8999 Acked 9998 Read 6 New Done Done false 9000 Acked 9999 Read 9 New Done Done false 9001 Acked 10000 Write 2 New Done Done false 9002 Acked 10001 Flush 0 New Done Done false 9003 Acked 10002 Read 5 New Done Done false 9004 Acked 10003 Flush 0 New Done Done false 9005 Acked 10004 Read 5 New Done Done false 9006 Acked 10005 Write 4 New Done Done false 9007 Acked 10006 Read 7 New Done Done false 9008 Acked 10007 Read 1 New Done Done false 9009 Acked 10008 Read 4 New Done Done false 9010 Acked 10009 Flush 0 New Done Done false 9011 Acked 10010 Flush 0 New Done Done false 9012 Acked 10011 Write 6 New Done Done false 9013 Acked 10012 Read 5 New Done Done false 9014 Acked 10013 Write 4 New Done Done false 9015 Acked 10014 Write 1 New Done Done false 9016 Acked 10015 Read 7 New Done Done false 9017 Acked 10016 Read 3 New Done Done false 9018 Acked 10017 Read 6 New Done Done false 9019 Acked 10018 Write 5 New Done Done false 9020 Acked 10019 Write 4 New Done Done false 9021 Acked 10020 Read 2 New Done Done false 9022 Acked 10021 Read 4 New Done Done false 9023 Acked 10022 Write 3 New Done Done false 9024 Acked 10023 Read 8 New Done Done false 9025 Acked 10024 Write 6 New Done Done false 9026 Acked 10025 Write 9 New Done Done false 9027 Acked 10026 Read 7 New Done Done false 9028 Acked 10027 Read 3 New Done Done false 9029 Acked 10028 Read 9 New Done Done false 9030 Acked 10029 Write 4 New Done Done false 9031 Acked 10030 Read 3 New Done Done false 9032 Acked 10031 Write 7 New Done Done false 9033 Acked 10032 Flush 0 New Done Done false 9034 Acked 10033 Flush 0 New Done Done false 9035 Acked 10034 Read 1 New Done Done false 9036 Acked 10035 Read 6 New Done Done false 9037 Acked 10036 Flush 0 New Done Done false 9038 Acked 10037 Read 4 New Done Done false 9039 Acked 10038 Read 2 New Done Done false 9040 Acked 10039 Write 7 New Done Done false 9041 Acked 10040 Read 1 New Done Done false 9042 Acked 10041 Read 10 New Done Done false 9043 Acked 10042 Read 7 New Done Done false 9044 Acked 10043 Read 5 New Done Done false 9045 Acked 10044 Read 9 New Done Done false 9046 Acked 10045 Write 1 New Done Done false 9047 Acked 10046 Write 5 New Done Done false 9048 Acked 10047 Write 2 New Done Done false 9049 Acked 10048 Read 1 New Done Done false 9050 Acked 10049 Write 3 New Done Done false 9051 Acked 10050 Flush 0 New Done Done false 9052 Acked 10051 Read 3 New Done Done false 9053 Acked 10052 Write 9 New Done Done false 9054 Acked 10053 Read 1 New Done Done false 9055 Acked 10054 Write 3 New Done Done false 9056 Acked 10055 Read 8 New Done Done false 9057 Acked 10056 Read 7 New Done Done false 9058 Acked 10057 Read 3 New Done Done false 9059 Acked 10058 Read 7 New Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 301 0 0 301 Sent 0 0 0 0 Done 0 301 301 602 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 9757 10050 10050 Downstairs last five completed: 9757 9756 9755 9754 9753 Upstairs last five completed: 9059 9058 9057 9056 9055 CLIENT: Up:0 ds:301 act:2 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:24.662738554Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:24.662837443Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:24.662865809Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:24.662902144Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:24.662923806Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:24.662946463Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:24.663133855Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} [8810] got start action [8810] I am going to restart Starting downstairs at port 8810 Make output file at "/tmp/dsc/downstairs-8810.txt" [8810][0] reports Starting {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:25.66507401Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:25.665138767Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:25.665166895Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-01-02T22:36:25.665193279Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:25.665213546Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] client re-new 302 jobs since flush 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:25.665254609Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"downstairs","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:25.665498168Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1215) [8810][0] reports Running {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.666379729Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.666563111Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"":"io task","client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.668618584Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.668656048Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"send last flush ID to this DS: 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.668680085Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"Replied this last flush ID: 9757","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.668986885Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.669020994Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.669044484Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.669066732Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} {"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:26.669088792Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1185,"client":"0","":"downstairs","session_id":"836679d4-4c63-42d3-928f-dc832a5b8247"} ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 10059 10058 10057 10056 10055 Upstairs last five completed: 9060 9059 9058 9057 9056 CLIENT: Up:0 ds:0 act:3 Replay: All jobs finished, all DS active. Test replay has completed Wrote out file "alan" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 10059 10058 10057 10056 10055 Upstairs last five completed: 9060 9059 9058 9057 9056 CLIENT: Up:0 ds:0 act:3 CLIENT: All jobs finished, all DS active. Replays:30 time: 6:02 Do final verify {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.349225178Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"6c4e859e6805a0246bf9e520f0708beaee75edc8\",\n git_commit_timestamp: \"2024-01-02T22:01:24.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.73.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"cc66ad468955717ab92600c770da8c1601a4ff33\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.349666827Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220} {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.349692867Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220} {"msg":"Crucible 0d740068-5979-4e21-aca5-6f3dc7bf90c6 has session id: cd25e34e-2445-4427-8cb6-8cc88b3dc290","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.349730047Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Upstairs opts: Upstairs UUID: 0d740068-5979-4e21-aca5-6f3dc7bf90c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.349774116Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Crucible stats registered with UUID: 0d740068-5979-4e21-aca5-6f3dc7bf90c6","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.349798662Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} Crucible runtime is spawned The guest has requested activation with gen:3 {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350059723Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"io task","client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35010589Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"io task","client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350132153Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"io task","client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35034767Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"io task","client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"0d740068-5979-4e21-aca5-6f3dc7bf90c6 active request set","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350402295Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350619624Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350675723Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350714141Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350780412Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"io task","client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.350822016Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"io task","client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.351454975Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.351496999Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.351525322Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.351548134Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.351570632Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.351592813Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35273272Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.352860409Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.353041522Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354136642Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354178212Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354202708Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354223591Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354246054Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35426845Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354294074Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354317457Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[0]R flush_numbers[0..12]: [976, 976, 973, 931, 966, 976, 967, 966, 973, 977, 924, 961]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354341233Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354364385Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354388314Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[1]R flush_numbers[0..12]: [976, 976, 973, 931, 966, 976, 967, 966, 973, 977, 924, 961]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354411752Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354434525Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354458491Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[2]R flush_numbers[0..12]: [976, 976, 973, 931, 966, 976, 967, 966, 973, 977, 924, 961]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354490559Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,The guest has finished waiting for activation with:3 "":"Wait for a query_work_queue command to finish before sending IO downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354538186Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35457247Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354595792Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354618132Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Next flush: 978","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354648973Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354689173Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"":"downstairs","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"No downstairs repair required","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354733047Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354769536Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"0","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35479525Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"1","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35481752Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"client":"2","":"downstairs","session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"All required repair work is completed","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354839673Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Set Downstairs and Upstairs active after repairs","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354868054Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"0d740068-5979-4e21-aca5-6f3dc7bf90c6 is now active with session: cd25e34e-2445-4427-8cb6-8cc88b3dc290","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.35490723Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} {"msg":"Set Active after no repair","v":0,"name":"crucible","level":30,"time":"2024-01-02T22:36:28.354947765Z","hostname":"ip-10-150-1-45.us-west-2.compute.internal","pid":1220,"session_id":"cd25e34e-2445-4427-8cb6-8cc88b3dc290"} Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576 Loading write count information from file "alan" Read and Verify all blocks (0..5000 range:false) Wrote out file "alan" Verify test completed Wrote out file "alan" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:5 States: Active Active Active GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY 49 Acked 1048 Read 100 Done Done Done false 50 Acked 1049 Read 100 Done Done Done false 51 Acked 1050 Read 100 Done Done Done false 52 Acked 1051 Read 100 Done Done Done false 53 Acked 1052 Read 100 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 5 5 5 15 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 1047 1047 1047 Downstairs last five completed: 1047 1046 1045 1044 1043 Upstairs last five completed: 53 52 51 50 49 CLIENT: Up:0 ds:5 act:3 ---------------------------------------------------------------- Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1053 1052 1051 1050 1049 Upstairs last five completed: 54 53 52 51 50 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program Jan 02 22:36:34.139 INFO accepted connection, remote_addr: 127.0.0.1:57790, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Shut it down Jan 02 22:36:34.140 INFO request completed, latency_us: 248, response_code[8810] Disable keep_running : 204[8810] Got stop action so:false kr:false , uri: /shutdown, method: POST, req_id: 90a00285-3cf6-446a-a159-8a824cfaf332, remote_addr: 127.0.0.1:57790, local_addr: 127.0.0.1:9998 [8830] Disable keep_running [8830] Got stop action so:false kr:false [8820] Disable keep_running [8820] Got stop action so:false kr:false [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit January 2, 2024 at 10:36:39 PM UTC Test ends with 0